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ABSTRACT 


This thesis presents a set of principles for web-based instruction based on 
literature from instructional design, usability engineering, and human-computer 
interaction. A questionnaire based on usability and instructional design attempts to show 
that in order to improve web-based instruction, usability and instructional design need to 
be taken into consideration when constructing long distance courses via the Web. The 
results show that usability and instructional design are dependent upon each other in 
order to present an effective on-line course while simultaneously learning from it. 
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I. INTRODUCTION 


A. PROBLEM STATEMENT 

Web-based and distance learning techniques are increasingly being offered as 
alternatives to conventional instruction. Since students now have more opportunities to 
complete or enhance their education via the web, techniques to attract and retain this 
large student population are needed. One advantage to asynchronous learning is that it 
allows class schedules to be flexible. Students are able to view class lectures and other 
instructional material any time of the day; whereas with conventional instruction, 
students must be at the specified classroom at the specified time. Another advantage of 
web-based instruction involves the simplicity of instructions and lectures via class notes 
and/or video presentation. These materials must be more concise and succinct due to the 
lack of face-to-face communication between students and professors. Conversely, these 
advantages can be disadvantages if proper principles are not followed because when 
students are self-paced, they may lose interest and not complete the course. This is also 
true with the simplification of course material. If not done correctly, students are not able 
to extract necessary information with which to understand complex concepts. 


B. MOTIVATION 

Techniques, for developing high-quahty web-based instruction are developed 
from three disciplines: (1) instructional design, (2) usability engineering, and (3) human- 
computer interaction. Instructional design principles are basic guidelines for organizing 
and presenting class material. The overall goal of constructing web-based instruction 
guidelines is to make an effective learning experience for students. 

Poorly designed computer interfaces may impair learning or impede learning 
progress and efficiency. Usability engineering impacts the quality of instruction because 
if students cannot use the system, they cannot learn. Likewise, professors cannot 
effectively present and instruct if they cannot use the system. The professor needs to 
structure information into the system, and the students need to take the information from 
the system. The computer cannot stand in the way when users try to get their work 
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completed. The system must support high productivity. This goal is achieved by 
designing interfaces that are consistent, controllable, and predictable, which makes it 
pleasant and effective to use (Schneiderman, 1998). In all, both parties should be able to 
use the system with relative ease. 

Human-computer interaction (HCI) guides the design principles and usability 
engineering by allowing hdividuals to communicate and interact with each other by 
means of the computer. In this case the computer is not only the “middle man” between 
the professor and the students, but also between students and students. If students, for 
example, spend more time trying to learn how the software operates rather than learning 
the course material, they lose valuable learning time, become frustrated with the system, 
and lose interest in the instruction. These considerations also apply to educators. 

The goal of this thesis is to develop a set of instructional design principles for 
web-based instruction and to evaluate those principles using existing evaluation methods. 
This thesis will then devise guidelines from the instructional design, human-computer 
interface, and usability engineering literatures and evaluate the guidelines with a 
measuring tool to evaluate on-line courses. 

It is not the intent of this thesis to find alternative methods of long distance 
education, but to seek out ways to supplement, complement, and enhance web-based 
education for students at NFS. 


C. RESEARCH QUESTIONS 

This research sets out to answer several key questions. First, which principles of 
instructional design apply to web-based instruction? Since web-based instruction is a 
fairly new process, there needs to be guidelines for instructors to use when putting a 
course online. This research intends to introduce a method of derived instructional 
guidelines from instructional principles for web-based instruction. 

This leads to the second research question: Which principles of general web-site 
usability apply to web-based instruction? In other words, what is the difference between 
web-based instruction vice general web interaction that makes web-based instruction 
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special? Users of the Internet equate web-based instruction and general web interaction as 
being the same when in reality, web-based instruction is taking a formal course via the 
web, and general web interaction is casual browsing on the Internet. A goal of this 
research is to clarify the differences of usability between web-based instruction and 
general web browsing. 

Third, which principles of human-computer interaction apply to web-based 
instruction? Computers are designed for various users. Certain users interact differently 
with the computer. Understanding human-computer interaction should improve web- 
based instruction better by helping designers understand how the average student learns 
via the web. The goal of this research is to clarify HCI principles for web-based 
instruction. 

Fourth, what are the typical capabilities and constraints associated with current 
web-based instruction software frameworks? When software developers make a long¬ 
distance education product, what constraints do they place on the course designer that 
might inhibit web-based instruction rather than assist students to learn? This research 
intends to identify constraints and capabilities of software products and how they affect 
web-based instruction. 

Fifth, what are valid measures of web-based instructional quality? Traditional 
instruction has been around longer than web-based instruction. Solid concrete guidelines 
have been established to measure the quality of instruction. Regarding web-based 
instruction, we do not know how to measure the quality of content for web-based 
instruction. A goal of this research is to find valid methods for measuring web-based 
instructional quality. 

Sixth, how are the data of valid measures of web-based instruction integrated into 
models to enhance the quality of instructional principles? We need to determine, if we are 
measuring navigational design, how well we can navigate the Internet and web-based 
site; or are we measuring the effectiveness of instructional design? Is there a tradeoff 
between navigational design and instructional design? A goal of this research is to 
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determine how navigational design and instructional design can be used to improve the 
quality of web-based instruction. 

D. ORGANIZATION OF THESIS 

This thesis is organized into the following chapters: 

Chapter I: Introduction. This chapter includes an introduction to the problem, 
motivation, and an outline for this thesis. 

Chapter II: Background. This chapter contains pertinent information regarding 
instructional design, human-computer interaction, and usability engineering. 

Chapter III: Theory. This chapter focuses on the theory of web-based instruction 
and how instructional design, human computer interaction, and usability engineering 
must be considered when developing a web-based instruction course. 

Chapter IV: Pilot Evaluation of Principles: This chapter summarizes a pilot 
project that initially tested the principles. 

Chapter V: Method, Results, and Analysis. This chapter discusses the method of 
testing, the results, and analysis of web-based instruction. 

Chapter VI: Results and analysis. This chapter discusses the results from the 

survey. 

Chapter VII: Conclusions and recommendations. This chapter will summarize the 
final results and thesis, and provide potential future work that can enhance the efficiency 
of the questionnaire 
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11. BACKGROUND 


A. INTRODUCTION 

In order to understand how instructional design and usability principles are 
intertwined in web-based education, it is important to understand each component 
separately. This chapter begins with an overview of human-computer interaction, 
usability, and instructional design, a discussion on why the areas are not independent of 
each other in web-based education, and what has been accomplished in web-based 
education with regards to instructional design and usability. 


B. HUMAN-COMPUTER INTERACTION 

Dix, Abend, Beale, and Finlay (1998) and Preece and others (1994) define 
human-computer interaction (HCI) as a “discipline concerned with the design, evaluation, 
and implementation of interactive computing systems for human use.” To produce good 
systems, HCI specialists attempt to understand how people effectively operate and use 
computer technology in their environment such as, the work place and home. They then 
develop techniques to assist designers in making computer systems that provide good 
interaction between the computer and humans (individual and group). The main emphasis 
of HCI research and design is to not only place the individual before the computer 
system, but also keep the individual in mind throughout the entire system design process 
because in the end, the user has to use the product to accomplish certain tasks. 

1. Disciplines that Contribute to HCI 

According to Preece and others (1994), 11 disciplines (Figure 1) contribute to 
HCI; Computer Science, Cognitive Psychology, Social and Organization Psychology, 
Ergonomics and Human Factors, Engineering, Design, Anthropology, Sociology, 
Philosophy, Linguistics, and Artificial Intelligence. Computer Science is about 
optimizing computer efficiency; Cognitive Psychology characterizes processes (e.g. 
perception, attention, learning, thinking, and problem solving) in terms of their 
capabilities and limitations; Social and Organizational Psychology studies the nature and 
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causes of human behavior in a social context; Ergonomics (or human factors) defines and 
designs tools and various artifacts for different work, leisure and domestic environments 
to suit the capacities and capabilities of users; Linguistics applies the scientific study of 
language; Artificial Intelligence (AI) is concerned with the design of intelligent computer 
programs, which simulate different aspects of intelligent human behavior; Philosophy, 
Sociology, and Anthropology are used to design and evaluate systems in order to provide 
a more accurate descriptbn of the interaction between users, their work, the technology 
they use in their environment; Engineering applies the science to produce an artifact; 
Design assists engineering by providing creativity. (Preece et al. 1994) 



Figure 1: 11 Disciplines of HCI From Preece et al 


To produce a system, three unique areas must be combined: 

• The 11 disciplines of HCI, 

• Tools developed by researchers and consultants based on those disciplines, 
and 

• participation from experts themselves. 
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Figure 2 shows how a design eyele normally operates. The knowledge and skills are used 
to develop eomputer tools. The design team uses the tools to assist in eonstrueting a 
system or software program. After the system is eonstrueted, an evaluation team will test 
the system and its usability. (Preeee et al. 1994) 
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2. Incorporating HCI into the Design Process 

When contending with HCI design issues, Nielson (1998) and Schneidermann 
(1998) state that a useful design philosophy for developing user-oriented human- 
computer interfaces is to consider the computer as a tool to aid the user in accomplishing 
tasks. A tool that requires more time, training, and effort to use than the task requires 
without the tool is a poorly designed system. Along the same lines, “Developers of 
software should not assume that they know what the users do because they are also users. 
This type of designing has resulted in poor, unusable systems that wastes hours of users’ 
time” (Preece, 2000, p. 112). In all, designers should begin to think like users, who do not 
care about how the computer operates, but how they use the computer to achieve specific 
goals. 

Incorporating HCI into the design process begins with support from management 
and input from team members. The teams discuss design tradeoffs that can 
simultaneously maximize requirements and user satisfaction. This is the foundation for 
the rest of project and is very critical to its success. Users of the system should be directly 
involved in the initial design process and during each testing phase. Knowing the user is 
also critical to successful system design. Developers should consult experts and 
researchers to understand human characteristics. After all, “users come in all shapes and 
sizes, with different personalities, abilities, experiences, resources, and needs” (Preece, 
2000, p. 124). A team of observers should visit the users’ work environment to observe 
and record the users’ daily activities and how they interact within their surroundings. This 
helps developers to understand the tasks that users will perform with the system in their 
environment. User interface guidelines should then be developed, documented, revised, 
and maintained for each project. This helps future projects because a different project 
team is able to research and understand previous work. When modifying a system, 
developers also have a base to build upon by understanding how to design within the 
varying constraints. System and software developers, analysts, management, and other 
developers should be trained often in human-computer interface design in order to 
increase their proficiency in their area of expertise. Training reinforces understanding and 
use of interface software tools that enhance the consistency of the interface and provide 
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an environment where interactive design is simple. It also supports program modularity, 
software libraries, development time, and cost. These aspects are critical to big projects 
because they can reduce risk while adhering to the development cycle schedule. Testing 
and prototyping early in the development cycle reveals flaws in interface requirements 
and user assumptions on how they will use the system. Testing design features should be 
performed on the intended user population. When changing and refining the design to 
users’ needs, developers should be flexible and patient. Early testing of the design 
maintains costs and helps the project to stay on schedule. Problems discovered during 
testing should be revised, corrected, then retested, and documented before continuing 
onto the next phase, milestone, and project completion. (Brown, 1991; Dix et al.l998; 
Hix & Hartson, 1993; Preece et al. 1994) 

A usability study determines how well designers, engineers, and developers 
incorporate HCI into their development of a system. This is where HCI envelops 
usability. HCI existed before the term “usability” came into use. HCI principles are 
typically based upon a user-centered design process, and certain design (human 
engineering) guidelines related to screen layout, use of fonts, color, highlighting, etc. 
Also, HCI concerns itself with the use of input devices, display devices, and their 
efficacy. Usability engineering, on the other hand, assisted in bringing the idea of a 
design metaphor into focus and placed greater emphasis upon building uncomplicated 
menu designs, meaningful control labels, and improving navigation to make systems 
easier to understand and use. Whereas usability deals with the process of building a 
system, HCI principles cover the entire design process of building a system. 


C. USABILITY 

Usability is a key issue in human-computer interaction; it is the principle 
commonly accepted to indicate the quality of a user interface (Paralangeli, Marchigiani, 
& Bagnar, 1999). Measuring the quality of the user experience when interacting with a 
system, such as a web site, software application, or any user-operated device is an 
example of usability. Nielson (1998) breaks usability into five characteristics; (1) Ease of 
learning; How fast can a new user sufficiently learn the program? (2) Efficiency of use; 
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Once the user learns the system, how fast can the user complete tasks? (3) Memorability, 
How effective can a previous user accomplish tasks with out relearning the system? (4) 
Error frequency and severity. How many errors occurred and how were they recovered? 
(5) Subjective satisfaction-. Is the user’s experience a positive one? All systems have all 
five characteristics of usability and all need to be considered in any design project. 


1. Elements Supporting Usability 

There are three distinct elements that support the usability characteristics. The 
first element is learnability, which is the ease with which new users can begin effective 
interaction and achieve maximal performance. It is measured by predictability, which 
measures past performance to improve future interaction. Synthesizability evaluates the 
effect of past operations on the current state. Familiarity pertains to how users apply their 
current knowledge and experience to new tasks for effective interaction. Sometimes 
“guessability” is when users learn something new by guessing on how to use it. If the 
user is familiar with a product and decides to use a similar product, the user is more likely 
to have a higher expectation, and is able to “guess” how the new product operates. 
Generalizability relates to how users extend their knowledge of specific behavior to new 
similar situations. Above all, consistency pertains to likeness in behavior arising from 
similar situations. This is usually accomplished through continuous tasks and keeping the 
tasks simple and constant. (Dix et al. 1998) 

The second element supporting usability is flexibility. Flexibility refers to the 
numerous ways users and the system exchange information. Dialog initiative allows users 
to communicate with the system within a constrained environment imposed by the 
system. An example is formatting text where users tell the system what font and style 
they want. Multithreading is another form of flexibility where the system supports 
numerous tasks at a time. Automatic teller machines (ATMs) effectively permit this type 
of behavior of user-system interaction. Task migratability relates to the transfer of control 
between system and users. The spell checker, for instance, displays a word for correction. 
After users decide to correct a word, they transfer control to the system for correction. 
Substitutivity refers to how the system can substitute values upon users’ requests such as, 
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changing margins or numeric values. Finally, customizability refers to how the users 
modify their system interfaces to their liking. Experienced users may want more features 
whereas novice users accept the standard software packages offered. (Dix et al. 1998) 

The final element supporting usability is robustness. Robustness is how the 
system supports users to achieve a set of goals. Observability affects robustness by 
allowing the user to evaluate how the internal system operates. Some examples are how 
well can users navigate, how well do the defaults perform for the users, and how well the 
system endures through constant use. Recoverability assists users in correcting errors 
where responsiveness is how fast the system can respond to users’ intentions. Finally, 
task conformance is how well the system supported the user’s goals. (Dix et al. 1998) 


2. Usability Standards 

Standards are usually set by national or international organizations to ensure 
compliance with a set of design rules by a large community. The Department of Defense 
(DoD) requires that software developers be certified according to the Software 
Engineering Institute (SEI) Capability Maturity Model (CMM). The minimal acceptance 
for a contractor is level three (defined process). At this level, the organization has a 
defined process definition and focus of their software development program. It includes a 
documented and implemented training program, integrated software management, 
software product engineering, and peer reviews. Generally, the defined process is 
supported by upper management and throughout the organization. 

The International Organization of Standardization (ISO) standard 9241 pertains to 
usability specifications for hardware and software development design. It gives three 
specific requirements for usability. The first part is the effectiveness, which is the 
accuracy and completeness with which users achieve certain goals. Examples of 
effectiveness include quality of solution and error rates. The second part is efficiency, 
which is how expended resources relate to the accuracy and completeness of achieved 
goals. Task completion time and learning time are measures of efficiency. Easily, 
satisfaction is how comfortable users feel when interacting with the system. Was the 

experience positive or negative? (Dix et al. 1998; Schneidermann, 1998) 
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3. Measuring Usability 

Usability testing ensures that all contractual requirements have been met, helps 
maximize the usability of the system by providing feedback during development, and 
provides evidence of testing in cases where legal issues may arise. “The greatest benefit 
about usability testing is that project teams can identify and correct errors, which can help 
to speed up delivery dates of projects while reducing project costs” (Schneidermann, 
1998, p. 128). Varying degrees of system errors are tolerated during testing. However, as 
the number of system inputs increase, the testing becomes more difficult. Varying 
degrees of system errors should be capped to ensure maximum usability of the system. 
Since lines of code are increasing within software programs, it is important that testing be 
increased and is more stringent in order to find, correct, and reduce the number or errors 
and bugs within the software development cycle. (Schneiderman, 1998) 

System design and tests should include expert reviewers who provide a 
comprehensive report on system problems and recommendations. Models of expert 
reviews are heuristic evaluations'. Expert reviewers critique an interface to determine 
conformance to a list of established rules. Guideline reviews'. Experts review the interface 
in accordance with the organizational guidelines. In consistency inspections, experts 
verify consistency across a family of interfaces, checking for consistency of terminology, 
color, layout, input and output formats. They also ensure that training materials and 
online help are also comprehensible for users when they use these services. Cognitive 
walkthroughs take place at each project completion phase and when the entire project 
terminates. After ensuring that requirements for a phase have been met, experts simulate 
users performing benchmarking tasks that physically test the requirements. This ensures 
that the interface is capable of carrying out common tasks. In a formal usability 
inspection, experts hold panel meetings. The individual responsible for the development 
of code discusses the code before a moderator and a panel of reviewers. Then the panel 
reviews the presentation and discusses the interface’s merits and weaknesses. These types 
of formal reviews are scheduled at project milestones, when experts are available, and 
when the project team is ready. These reviews are for critiquing the product not the 
individuals who have developed the code. The reviews help the development effort by 
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finding errors early during development. This ensures that at product completion, the 
product has no or very few errors, and is usable for the customer. If time does not allow 
for formal reviews, quick informal demonstrations of the product provide useful 
feedback. (Dix etal. 1998; Preece et al. 1994; Schneiderman, 1998) 

A usability study follows formal testing. During this phase, surveyors test a small 
portion of end users to determine how effective their products are. The end users are 
people who interact with the interface, and they determine how well the product could 
serve them in accomplishing their tasks. Some methods of a usability study are user 
ratings where the user rates a product based on a numeric value. Surveys also help 
developers test their products. Not only do users rate the product, but they also have an 
opportunity to write comments after each rating. A survey is more in depth than a rating 
because users explain why they gave that particular rating. Another method to conduct a 
usability study is through an interview. The interviewers can ask certain questions and 
write down responses to rate the usability of the product, and ask additional questions 
that might arise during an interview as well as observe and write down comments on how 
the user interacted with the system. In these sessions, the surveyors are observing the 
participants’ body movements and comments when they are initiating set of tasks. 
(Preece et al. 1994) 

D. INSTRUCTIONAL DESIGN 

This section is mainly based upon Charles Reigeluth’s book, Instructional-Design 
Theories and Models: An Overview of their Current Status 

In his presidential address to the American Psychological Association in 
1899, John Dewey (1900) called for the development of a “linking 
science” between learning theory and educational practiced (Reigeluth, 

1983, p. 5). 

Like a jigsaw puzzle, instructional science emphasizes how to put learning models 
and theories together. Instructional design (ID) links learning theory and educational 
practice. It is a body of knowledge that prescribes actions to optimize desired 
instructional outcomes such as achievement and affect (Reigeluth, 1983). Reiser (2001) 
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states that the field of ID and technology improves learning and performance by 
encompassing the analysis of learning and performance problems, design, development 
implementation, evaluation, and management of instructional and non-instructional 
processes. By understanding instruction and the five processes of instruction, one is able 
to understand how ID impacts instruction. 


1. Five Processes of Instruction 

First, instructional theory should not be mistaken for learning theory, which is 
concerned with what happens to the learner. Instructional theory is concerned with what 
the teacher does. For this reason, instruction is concerned with “how” to teach. Design 
falls under instruction because design relates to “what method will the instructor use to 
teach” (Reigeluth, 1983). Figure 3 shows how design relates to instruction in regards to 
the education process. 



Figure 3: Design Relationship Within the Education Process From Reigeluth 

ID involves understanding, improving, and applying methods of instruction. 
Educators are concerned with what methods of instruction are best for bringing changes 
in student knowledge and skills for a specific core content and student population. ID is a 
“blue print” for what method should be used for the course and students. Instructional 
Development takes the blue print and begins building a course upon the design through 
notes, lectures, and/or lesson plans. Instructional Implementation executes the lesson 
plan. Then Instructional Management maintains the lesson through adding and/or 
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deleting topics to keep it current. Finally, Instructional Evaluation is concerned with how 
effective the design was. Its main objective is to find weak areas to repair. (Reigeluth, 
1983) 

Of Instructional Design, Development, Implementation, Management, and 
Evaluation, ID has the most impact on the instruction process because it is the foundation 
for the development of instruction (see Figure 4). 



Figure 4: Relationship Between Design, Development, Implementation, 

Management, and Evaluation From Reigeluth 


The other areas are dependent on design as design is dependent on them for input. 
But the other areas are not dependent on one another. They may build upon one another, 
but they do not provide input into one another l ik e design does. Design inputs into 
Development by providing the initial plan to construct something such as web-based 
instruction. Second, Implementation relates to design by incorporating various methods 
of design to teach a course. Management manages various methods of design, and 
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Evaluation identifies and remedies weaknesses in the instructional system. (Reigeluth, 
1983) 

To be effective, design is dependent upon the other processes. Development 
provides economical information to the design process. Some designs are more expensive 
than others to develop and it is up to the administrators to decide what type of design to 
use. Implementation not only provides an economical decision (because some designs are 
costlier than the others to implement), but it is also used for constraints. For example, an 
innovative design may be economically sound, but may be poorly implemented. 
Management also provides economical input into the design process because if the design 
cannot be managed properly or it is costly to manage, the design is then rendered 
expensive and may have an economic impact over the long haul. Evaluation validates 
how well the instruction was designed, although it does not provide direct input into the 
models and theories of ID. (Reigeluth, 1983) 

2. Models And Theories Of Instructional Design 

To make instructional design more effective, instructors need to understand what 
complete set of strategies provides the best-desired outcome under given conditions or 
constraints. Reigeluth (1983) describes an instructional model as: 

An integral set of strategy components, such as: the particular way the 
content ideas are sequenced, the use of overviews and summaries, the use 
of examples, the use of practice, and the use of different strategies for 
motivating the students. 


Eike an architect’s blueprint showing many different designs and features of one 
type of building, instructional design should also show different views of instructions in 
order to achieve the desired outcome. Each view is a blueprint of what instruction should 
be like. 

These models may be fixed where the same method variable is prescribed 
regardless of the students’ actions. Or they may be adaptive where different method 
variables are prescribed according to the students’ actions. They then build upon a set of 
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principles that are referred to as instructional-design theory (or instructional theory). 
Instructional theory is a set of statements that take the form of: conditions-models- 
outcomes. (Reigeluth, 1983) 

3. Evaluating Instructional Design Theory 

When analyzing instructional design, two questions should be asked. Is it 
instructional or learning and how good is it? First, instructional design theory focuses on 
the methods of instruction and how to manipulate the instructional environment rather 
than on learning processes. Furthermore, is it a theory rather than a model or a list? Does 
the theory focus on the pattern of models-conditions-outcomes? For instance, does the 
theory have one or more models associated with it; does it also have a set of conditions 
under each model to be used; and does it have outcomes for each model under each set of 
conditions. (Reigeluth, 1983) 

After determining the theory, evaluating it for “how good it is” is the next step 
taken. Some major criteria to measure instructional theory are: 


Comprehensiveness. Does the theory include strategy components from all 
major classes of methods? Are the boundaries and limitations clearly 
explained? 

Optimality or usefulness. Are there better theories that can achieve desired 
outcomes? If not, analyze the weak areas on how they can be improved. If 
so, can they be incorporated into an existing theory to produce a better 
model? 

Breadth of application. Consider all models collectively rather than as an 
individual model. 

Parsimony. Is its degree of complexity warranted? Is it cost effective? 
(Reigeluth, 1983) 
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4. 


Instructional Media 


a. Why is the Instructional Media important for Instructional 
Designers? 

Instructional design is delivered through Instructional Media (IM). Reiser 
and Gagne (1983) define Instructional Media as “every physical means by which 
instruction is presented ta learners. It is every physical means of instructional delivery 
from the live instructor to the textbook to the computer and so on that would be classified 
as an instructional medium” (Reiser, 2001, p. 55). In this research, the instructional 
medium is the computer and the World Wide Web; or the Web for short. 

Previously, Phillips (1998) stated that the computer was used as a training 
tool to fill the void of distance learning. Designers created tools such as, computer-based 
instruction (CBI), computer-assisted instruction, or computer-based training (CBT), 
which provided interaction between learner and the content that met instructional goals. 
However, these tools were rigid due to their lack of openness, flexibility, and the learning 
environment was only between student and content. (Khan & Ealy, 2000) 

The Web, on the other hand, provides an open, dynamic, and flexible 
learning environment with implications for countless applications with respect to 
education and training. The Web allows instruction between learner and content, 
instruction between learner and students, and among the learners (Khan and Ealy, 2000). 
Also, E-Eearning via the Web is potentially cheaper and more productive and can be 
delivered with more timeliness than either classroom learning or traditional computer- 
enhanced teaching (Meyer, 2001). 

Due to the Web’s ability to allow people to communicate economically 
and relatively quickly with each other in nearly every area of the world, government 
agencies, corporations, and universities are taking advantage of this instructional medium 
by placing training packages and courses online. Corporations, such as American 
Skandia, use e-learning to train and reskill its workforce. Instead of spending days away 
from their desks sitting in a classroom, employees can log into the company’s training 
site and attend classes during breaks, lunch, and evenings at home at their own pace 
(Meyer, 2001). The American Iron and Steel Institute (AISI) believes that e-learning not 


18 



only saves money, but can also reach entry-level engineers who cannot obtain permission 
to travel to live seminars, to help them keep current and improve their engineering 
accreditation. Along the same lines, the American Institute of Architects (AIA) provides 
the same services for architects. Arizona State University, British University, and the 
University of Salford in the U.K. offer undergraduate as well as graduate programs for 
students who are working full time, raising families, and cannot attend classes until 10 
pm. When they access the learning sites, the students connect from Malaysia, Europe, or 
Australia. Overall, the Web provides the flexibility for learners to achieve a quality 
education at their own time and pace. (Rosenbaum, 2001) 

b. How Instructional Designers Use the Instructional Media 

Instructional designers should always take into account the person at the 
other end of the wire and how designers will use the technology to enhance instruction 
(Khan & Ealy, 2001; Pallof & Pratt, 1999). Eeamers may live in various parts of the 
country or world and their technology may vary. One student may have the most recent 
technology while the other is still using an old source. This may cause a problem for 
students due to the capabilities of each system. The student with the year 2001 
technology such as a Pentium 4, 1.4 GHz can download items and connect faster, 
whereas the student with the slower and older technology such as an X86 model will take 
longer to perform these same actions. 

In addition to these problems, instructors need to learn how to use the 
technology in order to deliver the subject material. Will the institution provide sufficient 
training for instructors to develop and deliver an online course? Will faculty also have 
trained support if a problem arises with the technology? 

Another consideration regarding users (instructors or students) is the 
amount of experience they may have with technology. Some users may be very adept and 
are able to catch on to the new technology while others may have difficulties with the 
technology. Eor designers, they have to take these two parties and their levels of expertise 
into consideration in order to produce a successful product. 
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When using courseware, software developed to deliver instructional 
material, the Web, CD-ROM programs, or video conferencing, instructional designers 
recommend that ease of use and visual appeal be taken into consideration. Ease of use is 
when the equipment is simple to operate for instructors and participants. It should be 
transparent where it helps the users accomplish their tasks. The hardware should provide 
enough power and memory to connect to the Internet and make navigating the course site 
easier. The best method to evaluate a system is evaluating how users comment on the 
system. Usually, the lack of comments represent that the users are satisfied with the 
system because it represents that transparency has been achieved. Visual appeal refers to 
the ability to create a site that captures users’ interests. Providing space for biographies 
allows participants to express their personalities and ideas. This assists instructors and 
students to fa mi liarize themselves with one another and strengthen the lines of 
communication by putting a face to the name and discussion(s). Virtual reality reduces 
the creation of a (potential) distorted image of participants. Video games have been doing 
this for years. The game Thief, developed by Looking Glass Studio Creations, for 
example, provides a brief tutorial on how to use the joystick or keyboard in order to 
maneuver successfully through the various levels of play. Players think that they are 
playing the game when in reality they are only learning how to use the game. When it is 
time to play the game, players are already familiar with joystick or keyboard movements, 
the rooms, the characters, and the various levels of the game. (Pallof & Pratt, 1999) 


E. USABILITY IN WEB-BASED EDUCATION 

Research in web-design and instructional design involves many aspects of 
usability. Nieslen (1998) refers to slow loading pages, obscure site design, and poor 
support for navigation as serious design flaws that hinder users from completing tasks 
(Evans, 2000). Hayes (1998) also states that courseware products are chosen based on 
technical specifications and features rather than on the usability attribute of ease of 
learning. This attribute should be for both users - the faculty who put the course online 
and the students who take the course via the web. She stresses that focus for course 
delivery systems is not given to user requirements and clear operational criteria are not 
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used. Decisions are often made based on the most recent and available technical 
information like bandwidth. Overall, Hayes discussed how usability principles can be 
applied to the web for evaluation, and how they are currently evaluated. Before her 
analysis of three groupware products, she took into account her target audience (faculty), 
their goals and tasks. Her analysis concludes that user feedback needs to be consistent 
and simple so users can easily create links to their files in a timely manner and reduce 
memory overload (Hayes, 2000). 

Parlangeli et al. (1998) discuss how multimedia systems in distance education 
affect learning. Their interest was in how learners learn to use the computer while 
learning. Even though multimedia systems are said to be the next major tool that “helps 
improve learning,” their results suggested otherwise. From three experiments dealing 
with heuristics and end-user evaluations, and a learner assessment based on a long 
distance class, the authors found that users mainly experienced navigation problems, 
memory overload, lack of information about the result of their own actions, and difficulty 
in performing the same actions across different environments. They concluded, “a 
difficult to use hypermedia system can negatively affect learning performance” 
(Parlangeli et al. 1998). 

Stayner and Procter (1999) deal with navigation, link time, and downloading 
while using the Web. Even though this study did not specifically deal with usability and 
educational design, it warrants consideration because of the research pertaining to 
navigation issues in instructional design. These factors may influence users’ choice of 
which links to follow. Sometimes, it is hard to anticipate arrival times and quality of 
documents when navigating the Web. Download times are unpredictable and long and 
quality is usually “bad” if users cannot make it to their destination. Conversely, users will 
wait for a long link if they know that the material is of good quality. Finally, users have a 
difficult time planning strategies because the Web is always unpredictable. In order to 
validate these assumptions, they tested two usability issues: 1. Content; How do users 
determine the quality of a document? 2. Temporal behavior of linking; How long do you 
wait for the link? They found that users looked for clues such as, country name, and 
common names from web sites to determine whether they should proceed navigating the 
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site, link to other sites for more information, and then they either downloaded items or 
left the site entirely. Furthermore, some users surfed the Web in the early morning or late 
at night when run times were better when less people access the Web. 

Jones and others (1999) attempted to test the usability of software used for 
education technology. The main concern is that the educational software is sacrificed for 
the sake of usability and HCI. They agree that usability and educational technology are 
dependent upon one another for a web-based course to be successful. However, they 
believe that the software evaluation process should have more input from educators in 
order to produce a better product for learners. Comparative studies have yielded 
information about what has occurred and not how learning has occurred. They believe 
that there is a need to understand “how the learning occurred” rather than “what has been 
learned” in order to produce better software for education. By studying the outcomes 
through interaction and outcomes, software developers and educators together can 
determine how usability contributes to educational goals. 

In other research regarding instructional design principles, Berge (1998), Najjar 
(1998), and Weston, Gandell, McAlpine, and Finkelstein (1999) mention usability issues 
such as, technical support, bandwidth, design, and navigation as principles for 
instructional design. These principles show how instructional design is dependent on an 
instructional medium like the Web, CD-ROM, or courseware, for its success. Without 
taking usability into account, educators sacrifice the quality of web-based education. The 
goal for these tools is to assist the users in presenting a course and taking a course. If 
these two goals cannot be met, then not only have designers failed by not taking usability 
issues into consideration, but also the institutions that purchase these tools fail because 
they have placed the burden on the instructors to deliver a quality education via the Web 
with a poor product. 
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III. INSTRUCTIONAL DESIGN AND USABILITY 

PRINCIPLES 


From a cognitive point of view, the most challenge [sic] is to bring 
together the educational content that students can access in both 
synchronous and asynchronous modes and to unify their respective 
pedagogical approaches. This merger is possible if an integrated 
educational approach is taken. In such an approach, the expositive 
teaching and active learning activities should be combined and balanced 
according with [sic] their respective pedagogical objectives, their didactic 
efficiencies, and technical constraints (Latchman, Salzmann, & Bouzekri. 
1999, p. 252). 


A. DERIVATION OF PRINCIPLES 

Squires and Preece (1996) argue that thinking of learning and usability as 
independent issues leads to superficial evaluations of educational reviews and that many 
teachers are not trained to consider usability (Jones et al. 1999). Usability and 
instructional design are different fields of study that are dependent upon each other for 
the success of web-based education. HCFusability is a procedure that is used to build 
effective systems while instructional design pertains to what the instruction should be l ik e 
(Reigeluth, 1983). Combining the two fields creates a good interface/system so that 
content can be easily accessed, read with ease, and examined with ease. Likewise, the 
content needs to be just as interesting in order to capture and hold the user’s attention 
regardless of how the system functions. For web-based instruction, Paralangeli et al. 
(1999) state that the application may offer relevant and useful information, but if the 
interface is difficult to use, the educational success of the system could be jeopardized. 
When interacting with [hypermedia] educational applications, the interface guides the 
student through an educational path. The student thus has to deal with a double learning 
process: on the one hand, s/he has to learn how to interact with the system. On the other 
hand, s/he has to acquire new and likely difficult concepts. These two aspects, namely 
learning how to interact with the system and learning the content it provides are not 
independent. If either of these balances is disrupted, students will more than likely have 
difficulties in learning via the web. Instructors may be able to provide good instruction 
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via the web, but if they are not trained to consider usability when designing a course, 
their course may be rendered ineffective because of a poor interface and usability. For 
these reasons, usability and instructional design must be taken into consideration when 
instructors put a course online. (Berge, 1998; Hayes, 2000; Jones et al. 1999; Paralangeli 
et al. 1999). 

Web-based instruction also depends upon the computer and web design for its 
success. In traditional education, instructors devise course objectives, print them, and 
distribute them to the students while explaining them. In web-based instruction, 
instructors devise course objectives and place them online. Instead of the instructor 
presenting and explaining the course, the computer now “converses” with the students. 
The computer acts as the interface between instructors and students. It does not translate 
“what is meant to be said,” rather the computer presents “what is being said” (Eisinger & 
Smith, 2000, p. 6). To assist instructors in this communication process between the 
instructor and student via the computer, instructional design principles were collected and 
derived from the literature. 

The literature review indicated that usability might substantially influence the 
quality and effectiveness of web-based instruction. The readings mentioned “technology 
minimalism” (Berge, 1998; Najjar, 1998) and “plan the lesson plan to the user’s ability 
skills” (Eisenger & Smith, 2000). Hayes (2000) even stated that despite published reports 
comparing online delivery systems, “none of these reports contain data about usability 
testing of the products reviewed. Even so, the phrase ‘ease of use’ is often sprinkled 
throughout the reports.” These patterns pertaining to both usability and instructional 
design appeared throughout the readings. 

Since traditional education is embedded in the education process, the best 
approach to web-based instruction was to compare similarities between traditional and 
web-based education. By breaking down a traditional course into very general parts, the 
following five areas were created: 

• Eirst day of class (welcome to the course). 

• This is what we are going to do (administrative). 

• Reading material and assignment (presenting the material). 
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Assess what has been learned (exam). 

Final course evaluation (assess the course). 


• 

B. FOUR REQUIREMENTS FOR AN EFFECTIVE ON-UINE COURSE 

1. The System Needs to be Reliable 

First, a system pertains to the software, the computer, the server, the amount of 
bandwidth, the IT (information technology) support to maintain the system, and the 
training to assist instructors in developing an on-line course. These components ensure 
the success of placing a course online. Second, a reliable system is outside of scope of 
this work because we (as educators) cannot tell students to buy the latest and greatest 
system. We have to hope that students do have a system that functions well. Whatever 
type of courseware the school purchases, the instructors and students alike are at its 
mercy. This is where institutions should consider the minimal requirements for web- 
based courses and the type of technology the users will use to access the course. A 
conservative estimate of the technology should be taken into account. Usually, the 
estimates should be the lowest acceptable technology on which the courseware is able to 
function. 

2. Tbe System Needs to be Usable to Instructors 

Instructors comment that time and the lack of training are the most common 
problems in putting an online course together. A checklist that assists instructors with 
putting a course online could alleviate the time and burden instructors have with putting a 
course online. A standardized checklist should be created to give instructors the essentials 
foundations in creating online courses. The difficult part is standardizing a generic 
checklist to help instructors, so that instructors should not have to be concerned about 
usability, HCI, and instructional design. Rather, they should be concerned with how to 
put the math, computer science, or history, course on-line. A checklist designed to cover 
these three areas would alleviate the problem. 
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3. The System Needs to be Engaging 

A good book will captivate a reader’s interest to finish reading the book. Well- 
designed eourses are no different from a good book. In traditional education, good 
courses are routinely ruined by poor presentation and organization. Students loose 
interest fast and cannot wait until the semester or quarter ends. In traditional education, 
the students can support each other through a peer network to help them get through this 
type of class. But in web-based education, since students are alone, they will turn off the 
system, and eventually drop the elass due to the frustration. To avoid this, the instruetor 
needs to present an organized and interesting leeture. A well-designed eourse should also 
engage the learner in aetive learning proeesses or aetivities. The learner should want to 
study due to the organization of the eourse and ease of aeeess to materials for learning. 

4. The System Needs to be Usable to Students 

Of the five eharacteristies that Nielson (1998) stated -- ease of learning, efficiency 
of use, memorability, error frequency and severity, and subjective satisfaction — ease of 
learning and efficiency of use relate most to web-based instruction. From the onset until 
the end of the eourse, students learn and use the system. Any system the students aeeess 
should be easy to learn and use. After overeoming the initial difficulties in learning, 
students should not have more problems when using the system. Memorability, error 
frequency and severity, and subjective satisfaction depend upon learning and using the 
system, whieh eome after time and through repetition of use. Only after students learn 
and use the system, then they can determine how “satisfied” they were with the system. 
Ease of learning and efficiency of use are important beeause they are needed to get the 
learner started and remain eonstant for the remainder of the eourse. 

This is not to assume that the student will neeessarily learn from a usable system, 
but that the system should help the student to aeeomplish intended goals of the eourse. 

These two eharaeteristics also pertain to instruetors beeause they have to put the 
course online. Sinee the emphasis of this thesis is placed on how the student ean rate a 
course with a measuring tool, the eharaeteristies will be diseussed in this portion of the 
thesis. 
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Ease of Learning 


Students learn an application while learning new material. It is important to take 
this into account because new and experienced users alike begin learning a system 
through trial and error (Hayes, 2000). This means that students are more apt to begin 
“clicking” features on the interface without reading the instruction manual or the “help” 
manual, which are normal and curious exploratory procedures that users possess. When 
users become confused or lost when completing a task, only then do they consult “help” 
(Nielsen, 2000). The courseware should provide the organizational structure for the 
course. The courseware needs to be simple and flexible in order for the students to learn 
the application and for the instructor to design the course. 


Efficiency of Use 


A system may be easy or difficult to use after learning how to use it. The users 
may have learned how to operate the system, but continued use of should not be difficult 
or counter-productive (Wyard & Churcher, 2000). If users have to continually struggle 
with a system while learning, users may develop a negative attitude towards distance 
learning - something that should be averted if ease of use is taken into consideration. 
Firewall access, bandwidth, navigation, and time to wait between changing pages are a 
few of the technological constraints that can render a web-based course ineffective. 
Users, regardless of experience, will not tolerate these problems that are associated with 
the technological medium. Students should be able to use the material with ease or very 
little difficulty regardless of user ability. (Berge, 1998; Hayes, 2000; Najjar, 1999; 
Weston et al. 1999) 
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C. INSTRUCTIONAL DESIGN AND USABILITY PRINCIPLES 

By combining the information from the literature, the similarities between web- 
based education and traditional instruction, and the four requirements of delivering a 
course, instructional design principles and usability principles were generated. Table 1 
shows the principles pertaining to web-based education. 


28 



Examples of Usability Principles Pertaining to Web-Based Instruction 


Present the material in simple, understandable paragraphs (simplification) 
Keep the content as organized and consistent as possible (consistency) 
Guide the students from one location to another (navigation) 


Examples of Instructional Design Principles Pertaining to Web-Based 

Instruction 


Get the learner started 
Present the subject material to the learner 
Let the learner assess the subject material 
Let the learner assess the overall course 
Let the learner assess the usability of the courseware 


Table 1: Instructional Design and Usability Principles 
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D. INSTRUCTIONAL DESIGN AND USABILTY PRINCIPLES DISCUSSION 


I. Usability Principles Pertaining to Web-Based Instruction 


Present the material in simple, understandable paragraphs (simplification) 


“Reducing complexity for users is still a very difficult aspect in interaction 
design” (Hix & Hartson, 1993, p. 35). After all, the computer is “lecturing” to the 
student and the student is reading from the computer. Eisinger and Smith (2000) state 
“the original thought behind online education was to take what was had and put it 
online”. A customization effort was then needed in order to give life to the material, 
which many people were not prepared to do (Eisinger & Smith, 2000). The content needs 
to be presented, so that the student does not become inundated with information. Taking 
into account that nobody can sit in front of a computer for long periods of time without 
interruption, dividing the learning experience into smaller, more manageable periods of 
time not only makes sense, but it is easier for a learner to schedule the lesson and retain 
information (Crawford, 1999; Eisinger & Smith, 2000). The time for reading text, for 
example, varies among the researchers. Some suggest 15 minutes and then an interactive 
exercise (Eatchman et al. 1999), or 20 minutes and then an example of the text (Berge, 
1998); but it should not be more than one-and-a-half hours [combine reading and 
exercises] (Eisinger & Smith, 2000). In another finding related to multimedia, “reading 
too much from the CD-ROM was tiring, so students printed some of the material and also 
requested that the instruction be in printed form” (Jones et al. 1999). In all, the material 
should not be presented like online books of the past because they are a “well-tried relic 
from the early days of the Internet” (Aussenhofer, 1999, p. 92) that were difficult to read. 
Educational material should not be. 
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Keep the content as organized and consistent as possible (consistency) 


This principle reinforces the simple and understandable paragraphs principle by 
organizing the content and ensuring its consistency for clarity. If the content is not 
properly organized from the beginning to the end of the course, this distracts students 
from learning the content because now, students are spending time on deciphering the 
information. “One of the more important rules in designing instruction is to have 
consistency among content, instructional objectives, and student practice, all leading to 
the evaluation that matches the practice on which students have been receiving feedback” 
(Berge, 1998, p. 73). Organized and consistent content also improves the chances of 
students comprehending and learning material. One method of ensuring good 
organization is “asking if content is relevant (value of content), reliable (content 
accuracy), and up to date (recovery)” (Weston et al. 1999, p.39). In traditional education, 
when instructors update their courses, they add, delete, and/or revise the content as 
needed. In web-based instruction, instructors should ensure that the content is neat and 
current because the content “communicates and teaches” students. After all, the content is 
the only link that students have to the instructor. 

Although the content may be clear, organized, and presented in understandable 
paragraphs, how is it being delivered? Attention to detail needs to be taken in great 
consideration when putting a class online because the instructor cannot correct 
himself/herself immediately after making a mistake (such as, grammar, spellings, etc.). 
The creativity of organizing the content can be a challenge because something so simple 
as a misspelled word can distract the student from learning (see Appendix A, section E, 
project 3: Early Analysis). In addition, strange fonts, lack of indentation, and spacing also 
cause problems for students. “It is important to ask if a legible font is used 
(typeface/fonts), whether numbering systems, headings, indentation, and spacing promote 
consistent presentation (format and layout)” (Weston et al. 1999 p. 39). 

Strategically placed hyperlinks also play an important role in affecting how the 
content is organized. Hyperlinks should be placed at the end of paragraphs after the 
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presentation of content. This ensures that students read the information first and then use 
the hyperlink to enhance the subject material. This method serves another purpose. For 
students who want to continue reading, they can continue the lesson without being 
distracted by a hyperlink in the middle of a paragraph (See Appendix A, section E, 
project five: Usability Analysis). For the students who want to enhance their knowledge 
base on that particular item, they are able to “explore information bases, discover 
relationships for themselves, or transform and organize information in ways compatible 
with their own needs” (Astleitner & Leutner, 1999, p. 8). Whether students click on the 
hyperlink or not, they have read the subject matter without interruptions and they have 
the choice to view or not view the content (Lund, 1997). 

Finally, the number of levels for linking to other sites should not matter if the sites 
have been organized properly. When Evans (2000), for example, developed guidelines 
for research-based web design guidelines, she found in one study that people were able to 
search for information faster and more accurately on bigger systems than a smaller 
system due to the site being designed to effectively guide users in finding information. 
The results of the study suggest that, “whether or not a website has been organized in a 
way meaningful to its users is also a key influence on its usability” (Evans, 2000, p. 305). 

Overall, the main focus of organization and consistency is presenting succinct and 
clear information, which focuses on cognitive and motivational design of hypermedia or 
web-based instruction (Astleitner & Leutner, 1999). After all, if the rontent is not 
organized, then the design of the system and the course is rendered useless (Hayes, 
1999). 


Guide the students from one location to another (navigation) 


Web-based instruction provides numerous benefits such as, quick and remote 
access to information, convenience, adaptability to change, and speed of communication 
(Piotrowski & Vodanovich, 2000). However, common design flaws such as, slow loading 
pages, obscure site design, and poor support for navigation renders the Internet unusable 
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due to users experiencing delays and running into dead links, which causes users to 
terminate a task prematurely (Evans, 2000; McKenzie, Mims, Bennet, & Waugh, 1999; 
Nielsen, 1993; Paralangeli et al. 1999; Piotrowski & Vodanovich, 2000; Stayner & 
Procter, 1999). These actions show that navigation is vital to the success of web-based 
education. Instructors should take this into account when they would like to have students 
visit other web sites. Navigating is the highway to information. Weston and others (1999) 
define navigation as, “how the student moves through the instruction and how the 
instruction is designed to facilitate understanding of organization and structure of 
content.” To prevent and/or minimize potential navigation problems, instructors should 
guide students through lessons by directing them where to go and what to expect. In 
other words, students open a door to a room (lesson objectives), walk around the room 
(read the lesson), and leave the room via a different door (perform an exercise or take an 
exam), and repeat the steps (proceed to another lesson). Informing users about the next 
site and time to wait to connect or download material, improves the Quality of Service 
(QOS) that students expect from a course. Sometimes connecting to another site takes 
time. Users should be able to tolerate delays in connecting because they have been 
informed about the time lag (Stayner & Procter, 1999). Once again, students should have 
the choice to continue go to the other web site or continue reading the lesson (Lund, 
1997). 


2. Instructional Design Principles Pertaining to Web-Based Instruction 

Get the learner started 


Students should initially meet the instructor and become fa mi liar with the course 
when they begin. The purpose and goals of the course must be clear from the beginning 
(Ross, 1999). They should be concise and brief. A description of the “big picture” of the 
course should be sufficient. The overall goal is to capture students’ attention while 
avoiding information overload (Crawford, 1993). Be creative in explaining the course. 
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According to Najjar (1999), “this type of instructional design should use a personal style 
(e.g. personal pronouns, names of specific people, direct quotations, vignettes of famous 
people) rather than a formal style to stimulate learner interest.” 

Letting the students feel welcomed may encourage them to become fa mi liar with 
the web site by exploring the course contents before taking the course (Ross, 1999). By 
the same token, allow time for an introductory lesson on how to use the web site allows 
students to relax and relieve the pressure of learning the computer and course at the same 
time (Astleitner & Leutner, 2000; Paralangeli et al. 1999; Weston et al. 1999). Using the 
video game Thief again as an example, by giving initial tutorials on how the game 
functions allows users to feel more relaxed while playing the game because they have 
already been exposed to how the game works. 

This area should also be the starting point and return point for students. Inform 
students that the Home page will be the gathering point for the class. Any new 
announcements or upcoming events will be posted in this area as the course progresses 
(Pallof & Pratt, 1999). 

Before continuing with the course, students should also know what is expected of 
them and what the instructor will provide. Usually in traditional education, the instructor 
explains administrative procedures such as, course objectives, syllabus or class schedule, 
and expectations on the first day of class in order to provide guidance and understanding 
for the duration of the course. These routine administrative tasks provide a starting line 
for students. In web-based instruction, these issues, along with how to navigate the course 
site, assist the student in becoming familiar and accustomed to the course site and 
technology. Finally, web-based instruction needs to begin with the end in mind. Course 
objectives should be established to determine what the students will learn and what skills 
and abilities they should take with them when they finish the course (Crawford, 1993; 
Pallof & Pratt, 1999; Ross, 1999; Weston et al. 1999) 

For an effective web-based syllabus, weekly topics for discussion and the due 
dates for graded assignments (quizzes, papers, tests, etc) should be placed in the syllabus 
section so that students understand how the class will meet objectives because “the 
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syllabus section is the road the students will take to reach their destination” (Pallof & 
Pratt, 1999, p. 88). The syllabus should not only be brief to avoid information overload, 
but it should also be topic-driven to give students more flexibility to use the school’s 
courseware. For example, if the syllabus contains a weekly topic discussion with 
readings, students have the freedom to read the material, asynchronously post their 
comments on the comment board, and participate synchronously and asynchronously in 
discussions before moving on to the following week. If a test is scheduled for that 
particular week, then students will also have the flexibility to take the test any time before 
the scheduled deadline (Latchman et al. 1999). 

The flexibility that web-based education provides can be abused if there are no 
defined expectations (Pallof & Pratt, 1999). Course expectations define the parameters or 
constraints that apply to successfully fulfilling course requirements. They also define 
what is and is not acceptable. In this section, student evaluation is explained. For 
instance, how much weight do assignments, tests, and group and individual participation 
hold? Expectations for participation should also be clearly defined to give students 
direction and let them know that synchronous and asynchronous participation is just as 
important as participating in traditional group participation. (Crawford, 1993; Pallof & 
Pratt, 1999) 

Expectations also include course conduct. Conduct is vital because students are 
more dependent on one another to complete assignments. If there is a team, for instance, 
and one team member is regularly absent from all team discussions, then the team may 
suffer due to the lack of participation of one individual. The rest of the team is dependent 
on this team member who they only know via a computer. Assuming that particular 
member hinders a group project, the group in general should not suffer due to one 
individual. Another example of an expectation is how many times a course can be missed 
before the student is dropped from the course or a grade reduced due to the lack of 
participation in discussion boards or coordinated chat groups for example. By the same 
token, students should take responsibility to notify the instructor when they will be 
unavailable. (Pallof & Pratt, 1999) 
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Along the same lines, the instructor should define his/her role in the course. Will 
the instructor organize and present the material, or actively engage the students by 
participating in group chats and discussions, or will the instructor be a cheerleader, letting 
the students teach themselves and only help when it is truly needed? The instructor’s role 
should be clear for students to understand the class organization and how it will operate. 


Present the subject material to the learner 


If the materials are not presented properly, students lose interest. “It is difficult to 
attend a boring class and even more difficult to attend a boring web site that contains a 
boring class” (Ross, 1999, p. 3). At the onset, students understand they are about to take 
the course and begin learning. At this juncture, “instructors shape the course by providing 
a content-rich resource, which encourages students to explore new avenues of learning” 
(Latchman et al. 1999, p. 248). 

How instructors organize the text influences how well students learn. Najjar 
(1998) recommends combining graphics with the information to reinforce the content and 
to focus students’ attention on the subject. To reinforce the metaphor, “a pictures tells a 
thousand words”, students take time to interpret what has been read through pictures. Not 
only are text and graphics easiest to use for on-line environments, but also when they 
match, students then have a better chance at learning the material (Berge, 1999; Najjar, 

1998) . 

Another consideration for placing text on-line is how students read web pages. 
Hayes (1999) reports that people do not read web pages as they would print-based 
materials. Users prefer to scan text rather than read it word for word on the Web (Hayes, 

1999) . As a result, the overall principles really need to be taken into account when 
placing the text. For example, after giving a brief review with graphics on the human 
nervous system, place a web link at the end of the paragraph for the students to explore 
more. Viewing the web links can be either mandatory or voluntary depending on how 
much information the instructor wants to present. 
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As in a traditional environment, instructors should encourage students to be 
interactive within the Web community. Kearshey (1997) states that, “the single most 
successful elements of on-line education is interaction among participants” (Berge, 1999; 
Weston et al. 1999). Depending on the instructor, this method can be synchronous (chat 
rooms) or asynchronous (discussion boards) or both. Engaging the learner through 
incorporating activities such as, practice and feedback, opportunities for reflection, and 
problem solving, greatly enhances the desired learning. Interaction provides an outlet for 
all students to voice their opinions (Astleitner & Leutner, 2000). It is also good for 
collaborative learning, since many students are more comfortable with contributing to a 
discussion on their own terms, with ample time for reflective responses, rather than to 
being put on the spot in the live traditional classroom (Latchman et al. 1999; Ciavarelli, 
2001). With on-line learning, the shyer students now can express their opinion without 
being stifled by the more expressive students. 

Interaction also helps students to develop their logic and thought processes by 
constructing through rehearsal, argument, and persuasion on discussion boards or in chat 
rooms (Berge, 1999). Instructors should have the students put their thoughts on the 
discussion board for other students to see and for peers to comment on. Along the same 
lines, Jones and others (1999) affirm that through interaction, writing messages help 
clarify students’ thoughts by developing arguments and building upon other students’ 
contributions. 

Bosco (1986) and Fletcher (1989, 1990) examined 75 learning studies and found 
that participants learned material faster and had better attitudes toward learning the 
material when they learned in an interactive environment (Najjar, 1998). When 
presenting materials, instructors should stress interaction and make their lessons more 
topic and discussion driven rather than just having students clicking on the site, reading 
the notes, and then clicking off the site (Berge, 1998). 

Instructors should take into account the time factor and information overload 
when they present materials. Students cannot participate effectively in chat rooms and 
homework assignments due to time constraints and the amount of work given. The 

amount of reading, discussions, and assignments should be distributed evenly for students 
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and the amount the student should give to the instructor should not overwhelm the 
student. Information overload inhibits students from learning; thus information should be 
given and performed in understandable paragraphs and within a reasonable time frame 
(Crawford, 1999; Ciavarelli, 1999). 

Let the learner assess the subject material 


This area demonstrates how well the students comprehended the subject matter. 
The instructor should divide the course between quizzes, tests, term papers, individual 
participation, group participation, and/or project assignment(s). Najjar (1999) says that, 
improving student learning performance results from the test matching the information 
that was learned and the given test matching the expected test. When the instructor 
decides on the type of assessment, it must match what was taught in the course. Like 
traditional education, students do not enjoy being told that the exam will be on subject x 
and the instructor gives an exam on subject y; or the lesson was on identification of body 
parts and the test is about the functions of body parts. In all, give the students what they 
expect because they cannot ask the instructor immediately via the computer as they can in 
traditional education. (Weston et al. 1999) 


Let the learner assess the overall course 


Feedback is essential for improving a poorly designed course. The main concern 
should be how the students experienced the course, the instruction, and the online 
experience. Giving students a means to respond via e-mail or a questionnaire builds a 
stronger, cohesive on-line community, since the students have more input and feel 
“equal” with the instructor (Astleitner & Leutner, 2000; Crawford, 1999; Jones et al. 
1999; Pallof & Pratt, 1999). In a traditional setting, students see the teachers, and make 
personal comments about the instructor when evaluating. They make it personal. In web- 
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based instruction, the evaluation should reflect the course material rather than the 
personality, resulting in better feedback for the instructor to enhance the course for the 
next group of students (Astleitner & Leutner, 2000; Pallof & Pratt, 1999; Rockwell, 
Furgason, & Marx, 2000). Furthermore, on-line evaluations may be more effective 
because the quiet students may contribute more anonymously without intimidation. 
(Berge, 1999; Pallof & Pratt, 1999) 


Let the learner assess the usability of the courseware 


The instructor should have the students provide evaluations at the end of the 
course to determine the ease of access to the program, delivery of the material, and 
technical support. The technology should meet the students’ needs because the 
functionality of the system is just as important as the design. (Hayes, 2000; Jones et al. 
1999; Lund, 1997; Pallof & Pratt, 1999; Ross, 2000; Weston et al. 1999; Wyard & 
Churcher, 2000) 

The faculty should evaluate the technology as well to determine how well it 
worked for them. They should also evaluate for additional training. They should critique 
how well the institution provided assistance to the instructors. The instructors cannot put 
a class online without assistance. If the institution is determined to have successful web- 
based education courses, the institution should ensure that instructors receive the support 
and training. Some forms of assistance are online help from the vendors of the 
courseware, tutorials, guidelines from trainers, and workshops (Hayes, 2000; McKenzie 
et al. 1999; Pallof & Pratt, 1999; Piotrowski & Vodanovich, 2000; Rockwell et al. 2000; 
Weston et al. 1999). 
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IV. 


PILOT EVALUATION OF PRINCIPLES 


To test the principles, a pilot test was conducted. The goal of the test was to find 
and correct deficiencies in the principles. The project began by identifying and defining 
the problem. The Naval Postgraduate School (NPS) has been actively developing new 
online courses, but lacks an effective tool to measure its long distance program. The 
solution was to create an online course based on the principles, and test its effectiveness. 
Instructors and students were the intended audience. Appendix A contains the project in 
its entirety. This section only provides a brief overview of the project. 

A. PROJECT SUMMARY 

After conducting the needs analysis, user analysis, and task analysis, requirements 
for designing and building the course were gathered for the design. Using paper, crayons, 
and pencil, a design of various interfaces with subject content was made, and then shown 
to various subjects for their critique. The course was a tutorial on using various Internet 
middleware. The tutorial had a page for each welcome, administrative, course objectives, 
syllabus, and assignment sections. The learning lesson was also included. 

When this design was shown to potential users, they commented on how the 
navigation buttons should have been placed in a different area rather than on commenting 
on the content of the course. As a result, various questbns on usability of the interface 
were asked to determine how it affected usability and learning. After making corrections 
based on the peer review, the design was transferred to the school’s courseware for a live 
demonstration. 

The criteria for evaluating the design was based on how well the students could 
navigate the site by following instructions. Since the school already had a proven and 
usable system that could deliver the course, the researchers decided to test the content of 
the course to see if the design also affected how students learn. 
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B. METHOD OF EVALUATION 

Four subjects took a web-based course tutorial regarding Internet middleware. 
The subjects had at least intermediate computer experience. Four students belonged to 
Computer Science, Information Technology, Systems Management, and Operations 
Research curricula. The subjects had a strong background in using computers and one 
student was also taking an online course delivered via a different courseware tool that 
was not used in the research. 

Since the course material could only be evaluated through subjective means, the 
research placed emphasis on collecting data on following course instructions, mainly 
through navigation. The subjects were encouraged to talk and think out loud. As they 
took the course, they talked and thought out loud so the researchers could subjectively 
analyze how the subjects’ understood the course content. The course time was 30 
minutes. During this time, the researchers wrote down their observations, the subjects’ 
comments, and actions as they navigated the course. When they finished the course, the 
researchers spent 20 minutes with the subjects to evaluate the course via a checklist. The 
checklist acted as a survey or post questionnaire. After each interview, the researchers 
and each subject evaluated and reviewed the checklist to obtain better results. The results 
were based on how they viewed the course when they were taking the course. Another 
reason why the researchers walked the users through the questionnaire was because it 
stimulated dialogue on what the users thought of the course. This was better than just 
“checking the boxes”. Furthermore, It assisted them in commenting on their experience. 
All in all, they also felt more relaxed during the dialogue, which provided the researchers 
with valuable feedback. After each session, the results from each researcher were 
discussed and compared. One researcher usually recorded a comment or behavior that the 
other researcher did not notice during the interviews. 


C. LESSONS LEARNED FROM THE INITIAL EVALUATION 

I. Instructional Design and Usability are Dependent Upon One Another 

As each subject took the course, they commented not only on the contextual 
errors such as, misspellings, grammatical errors, lack of font and indentation, but also on 
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the brightness of the screen, the button colors were to bright or dark, and the navigation 
was difficult, especially when the subjects left the tutorial and visited the web sites. The 
students did not know when to stop navigating and return to the tutorial. Even though the 
emphasis of the evaluation was placed on the presentation method, usability issues crept 
into the project, which reinforced the belief that usability and instructional design are 
dependent upon one another. 

2. Flaws in the Questionnaire’s First Draft 

The students had difficulties in understanding the questionnaire due to the poor 
wording of questions. They did not accurately reflect what was intended for the research, 
since the subjects were repeatedly told to perform various actions again and the subjects 
also commented on how to interpret the questions for answering. 

Another mistake the questionnaire contained was the rating system. The 
researchers did not have experience and skills required to develop a rating system To 
compensate for the lack of experience, the researchers used phrases such as; ‘1 = strongly 
disagree’, ‘2 = disagree’, ‘3 = no idea, not really’, ‘4 = agree’, ‘5 = strongly agree, I feel 
great’. The wording confused the subjects because they did not understand ‘no idea, not 
really’. Moreover, no previous method for quantifying ‘no idea, not really’ existed. As a 
result, the initial rating system distorted the accuracy of the data due to using an unproven 
rating system. 

D. QUESTIONNAIRE REVISION 

I. Sections and Wording Revision 

The instructional design sections were shortened from five sections to four parts. 
The Administrative and Welcome section were combined into the Getting the Learner 
Started section in order to provide greater clarity for the users. Also the wording within 
the questionnaire went through numerous drafts. After each revision, an informal review 
consisting of NFS students who had taken web-based instruction and NFS instructors 
who have given online courses, reviewed the questionnaire for more revisions. Due to 
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their input, the questionnaire evolved into a more-focused and accurate tool. Appendix B 
contains the revised and completed questionnaire. 

2. Replacement of the Rating System 

To make the instrument accurately reflect the measurement dimensions, the 
Questionnaire Interaction Satisfaction (QUIS) version 7.0, developed at the University of 
Maryland by a team of researchers in the Human Computer-Interaction Interaction Lab to 
measure users’ subjective satisfaction with specific aspects of the human-computer 
interface, was adopted. This particular questionnaire addresses reliability and validity 
problems found in other satisfaction measures, and is highly reliable across different 
spectrums of interfaces (Norman, Schneiderman, Harper, & Slaughter, 1998). Appendix 
C provides a deeper insight into the QUIS survey. 

The flexibility of QUIS was also a factor in using the questionnaire because there 
were many sections that the project did not use and they were easily omitted from the 
Instructional Design - Usability Principles questionnaire. The QUIS dealt specifically 
with the system and delivery, whereas the Instructional Design - Usability Principles 
questionnaire also needed to specifically focus on instructional design issues. Adopting 
the QUIS format provided structure for the Instructional Design - Usability Principles 
questionnaire provided this flexibility. 

Another appealing aspect of the QUIS questionnaire was the 9-point rating scale. 
The scale goes from one extreme such as, ‘illogicar to ‘logical’. The midpoint of the 
rating scale (5) can be used as a criterion representing an average. Above 5 is considered 
being better than normal, and below 5 is considered less than normal. By plotting the 
results and finding the mean, the reliability of each variable can be determined as well as 
identifying flaws in the system and the questionnaire. If there are unusual rankings, the 
graphing will assist in identifying areas for analysis. (Norman et al. 1998) 
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V. METHOD OF EVALUATION 


A trial evaluation was conducted to evaluate the revised Instructional Design- 
Usability questionnaire as an effective measuring tool for web-based instruction. The 
subject pool consisted of resident and non-resident students from three classes using the 
Web as a mode of instruction. 

A. SUBJECTS 

The first class, SS3011, Space Systems Technology and Applications, was a 
purely web-based course involving students from NFS and throughout the United States. 
The second course, OC2022, Scientific Fortran Programming, consisting of five students, 
was a web-enhanced course, where students went to classes and used the web as an 
enhancement. The third class, MN3384, Acquisition Production and Quality 
Management, contained two sections. The first section consisted of resident students and 
the second section was delivered via Video Teleconferencing Environment (VTE) to 
students in Huntsville, AE and Eort Monmouth, NJ as well as resident students. The two 
sections were also web-enhanced. Of the 48 students from both MN3384 and OC2022 
who attended lecture and used the Web for laboratories, tests, and homework, 28 students 
took the survey. 

B. INSTRUMENT 

The Instructional Design-Us ability questionnaire consisted of 83 questions 
involving questions based on the background material and derived principles involving 
instructional design, usability, and web design. Two questions related to demographics in 
order to identify the classes and how the class was presented. The rest of the questions 
pertained to the questions regarding usability and instructional design. Appendix D 
provides the questionnaire used in the survey. 
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C. PROCEDURES 

1. Administration 

The questionnaire was administered via the Web in June, 2001 to the students. 
The personal information gathered from each student was limited to the name of the class 
and if the class was delivered via purely web-based or hybrid. There was no other 
personal information collected that could potentially relate the participant to his/her 
responses. 

2. Preparation and Evaluation 

Because of Is length, the time to take the questionnaire was estimated to be 
between 20-30 minutes. A brief explanation was given to the students regarding the 
importance of the survey and asked for their honest opinion about how the course was 
presented. The questionnaire was filled out on a voluntary basis. Twenty-eight (28) 
students responded. 

3. Medium 

The NPS Office of Strategic Planning, Educational Assessment and Institutional 
Research (SPEAR) converted the paper format of the questionnaire into a web-based 
survey through the software Survey Said. The software creates and processes surveys 
online. Once the questionnaire was placed online, the students were sent the URE address 
to take the survey. The answers were recorded into a database and interpreted using an 
Excel spreadsheet, and then graphed by conditions using the program Stat View. 
Responses with ‘ Not Applicable’ were omitted from the calculations to provide accurate 
data for the average. The results are tabulated in appendix D. 

4. Evaluation of the Questionnaire 

The questionnaire was evaluated in part by how the students rated the course. The 
intent was to demonstrate the derivations of the average means from each class above and 
below a criterion, and then make an analysis of how the courses were rated. Since the 
rating scale is based on the QUIS 7.0 system of 1 to 9 (APPENDIX C: 
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QUESTIONNAIRE EOR USER INTERACTION SATISEACTION), the midpoint (5) 
was the criterion for a neutral rating. If the rating was above 5, then the rating can be 
considered above average and better. If the rating is below 5, then the rating could be 
considered negative if the questions written to be positive. Eor instance, in part four of 
the questionnaire, a question such as “there were difficulties with navigation?” is not 
negative if the rating is a 2. Regardless of rating, the questionnaire attempted to reveal 
how groups rate particular parts of an online course. 
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VI. 


RESULTS AND DISCUSSION 


A. RESULTS 

Appendix D, section A shows the questionnaire used for the survey and section B 
contains comparison graphs between the two groups. Graph 1 shows the average mean of 
each section of the questionnaire. Graphs 2-5 show the cell mean of each question from 
each section of the questionnaire. 

The results from the questionnaire (Appendix D, graph 1) indicate that the 
VTE/online group enjoyed a higher degree of user satisfaction than the hybrid group. The 
standard deviations between the two groups were between 1.5 for the first three sections, 
and 2.5 in part 4. In sections 1 to 3, both groups rated the course positively. However, in 
part 4, the hybrid group rated the usability of the courseware more pessimistically than 
the VTE/online group. 

1. Part One: Getting the Learner Started 

Graph 2 shows that the hybrid group had more difficulty in understanding course 
concepts. The average cell mean difference from questions 4-12 was 2. The class syllabus 
and course expectations were not as clear for the hybrid group as they were for the VTE 
group. Both groups agreed in question 13 that the content from the web site was 
meaningfully labeled. However, both groups differed again on how the web designs of 
the class syllabus and course expectations were presented on the course site. 

2. Part Two: Presentation of the Subject Material 

Graph 3 shows that both groups were consistent in their ratings of questions 21- 
42. Despite some exceptions in questions 29-30 and questions 35-38, where the VTE 
students rated portions of the course slightly lower than the hybrid group, both groups 
answered positively on the instructional quality of the course. However, questions 44-51 
show the ratings dropping and fluctuating between both groups where the VTE students 
rated lower than the hybrid group and then returning to consistency in questions 51-53. 
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3. Part Three: Assessing the Subject Material 

In graph 4, both groups rated consistently high with the exceptions of question 59, 
which had a cell mean of 5 for the hybrid group and 6 for the VTE group, and question 
61, which had a cell mean of 3 for the hybrid group and 1 for the VTE group. 


4. Part Four: Assessing the Usability of the Courseware 

In graph 5, the VTE students rated higher than the hybrid group in questions 68- 
71. However, qaestions 72-80 show the VTE group cell mean average dropped to 2, 
whereas the cell mean average for the hybrid students dropped to 4.5. 


5. Conclusions 

The questionnaire displayed its potential to be used as a checklist for instructors. 
In areas such as question 5 (appendix D, graph 2), upon review, the instructor understood 
why both groups rated the question relatively low. This is an example of how the rating 
scale and the anchoring words provided solid measuring support to reveal positive areas 
as well as negative areas. 

Despite the positive areas, the questionnaire also revealed a limitation by not 
providing evaluation items for the questionnaire itself. There was no mechanism added 
into the questionnaire to determine if the questionnaire is an effective measuring tool. 
Since numbers can be interpreted many ways, there needed to be a section asking the 
students about the tool itself. 

Eurthermore, the volume of low responses indicated another limitation. This 
questionnaire would have been more effective if there were more participants. One class 
failed to take the questionnaire due to an administrative error, and the number of students 
from another class was too low to use as a case study. In the third course, MN3384, 24 
students responded, which provided a good assessment of the course and how it could be 
improved. 


50 



Since the students came from either resident web-enhanced (hybrid) or non¬ 
resident (VTE web-enhanced), the responses provided a means of categorizing and 
analyzing the results based on these two conditions. 

B. DISCUSSION 

The data collected during the course of this study reinforced the instructor’s 
opinion on problems that he encountered with his web site during the course. In a post¬ 
survey interview, trends from selected questions were discussed to determine what were 
the causes and solutions to the responses. 

The results in graph 1 (Appendix D) show that the VTE students had a higher 
level of user satisfaction throughout all areas of the questionnaire. This could be 
attributed to the instructor not being physically present, thus forcing the non-resident to 
concentrate more on the material than the resident students. Eurthermore, the VTE 
students currently work in the field of acquisition and were already fa mi liar with the 
lecture material from the course. As a result, they could easily apply lectures and 
assignments to a daily routine whereas the resident students were in the process of 
learning new material and lacked the work experience. 

The quotes from the discussion represent the thoughts from the instructor on how 
he viewed the ratings of the two groups. 

I. Part One: Getting the Learner Started 

The VTE students felt more comfortable in getting started with the course due to 
initially understanding the course topic better than the resident students. There was a 
difference of opinion between the groups in the value of course navigation instructions 
and biography of the instructor. The VTE students felt that having this knowledge was 
helpful for them. This extra information substituted for the lack of the instructor being 
physically present. The students understood the instructor and pertinent information on 
navigation. The hybrid students, on the other hand, felt that this information was not 
pertinent to them due to the instructor being physically accessible at all times. Whenever 


51 



a problem arose, the resident students could schedule visits and talk to the instructor 
before and after class. Through informal discussions, the hybrid students developed a 
better relationship with the instructor. 

Despite both groups agreeing that the content from this section was meaningfully 
labeled, they differed again when rating the content in course syllabus and course 
expectations. “The content was not provided in small paragraphs, the color of the pages 
was just black and white and nothing in between. They were not labeled well. For 
example, there was information on grading and assignments but there was not an 
explanation regarding the course itself. The information was scattered and not placed into 
modules. Now there are objectives and explanations placed before the start of each 
module and reading material.” 

Since the VTE students probably had taken web-based courses previously, it is 
more than likely that they have already seen better or worse course sites. They also take a 
more “we will get through this philosophical attitude.” They understood and accepted 
that there was a distance gap and the best method of success was to consult with one 
another when problems arose or if the material was difficult. For the resident students, 
this was a new experience for them. Learning new subject matter takes time. The 
vocabulary, theories, and background are difficult concepts to grasp initially. After the 
first test or assignment, students begin to feel comfortable because they have established 
some foundation. The resident students also had three other classes to worry about 
whereas the VTE students only had this one. 

2. Part Two: Presentation of the Subject Material 

The trend from each group was consistent with what the instructor did not do 
during the course. Eor example, the instructor did not use the discussion board and chat 
room to enhance the Web portion of the course. The instructor incorporated these 
changes for the next course. He will use the discussion board as a tool to answer 
frequently asked questions (EAQ) and make available a student self-assessment from 
each lesson objective where the students are able to discuss objectives to provide a 
“better opportunity to grasp the material for both sections.” The instructor would like to 
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see direct and succinct comments ranging from one-quarter of a page to a one-half page 
relating to the assignment. He believes that the discussion will not only enhance the 
students’ writing skills and thought processes, but also “assist the bashful students in 
being vocal.” 

In addition to the discussion board, the instructor has adapted the chat room for 
students to share their ideas through interaction. He would like to place students into 
small groups, consisting of a mix of resident and long-distance students to interact and 
learn from each other. The instructor believes that this mix combines the strengths of the 
two sections. The resident students are the users and experimenters of the material while 
the non-resident students are the developers who possess the “real-world knowledge” 
they can share with their resident peers. “The cooperation between the two groups should 
equal success.” 

Incorporating these two methods of web-based instruction will provide the 
necessary feedback to facilitate a successful online course. The instructor feels that he 
should receive feedback faster to adjust to the concerns and needs of the students. “These 
are new methods of instruction and we [instructors] should use these tools to help us pass 
on the knowledge to the students who can effectively apply it when they return to their 
jobs.” 


3. Part Three: Assessing the Subject Material 

The students from both sections did not have a mechanism for submitting a 
critique of the exams to provide feedback to the instructor. Also, there was a lack of 
feedback given back to the students to review incorrect answers from the exams. In the 
newly revised course, the instructor has an area for students to comment not only on 
exams, but also on written and reading assignments. The area is anonymous for students 
to “entice students to speak freely without retribution.” In all, the instructor does not care 
who gives constructive criticism, but that there is a way to receive the advice. 

Finally, the instructor would like to make available for the non-resident students a 
means for them to receive feedback on the final exams. The resident students have the 
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luxury of going to the instructor’s office to inquire about the their performance and 
receive additional feedback on single questions. The non-resident students, on the other 
hand, have no means of physically visiting the instructor. This situation can be more 
difficult and frustrating for the non-resident students if they desire to obtain feedback. 
The instructor would also like to afford an area for the students to have access to review 
questions as they study for the exam. 

4. Part Four: Assessing the Usability of the Courseware 

The VTE students experienced fewer problems when using their computers to 
access the web site. This is significant due to the fact that during the course, the original 
web site was not accessible to due to firewall problems to a few resident and non-resident 
students. As a result, the instructor hastily constructed a basic web page consisting of 
assignment folders for each week. The resident students preferred the simple web page 
while the VTE students continued to access the original web site. 

Another reason why the VTE students had better user satisfaction was because of 
their system. The VTE students used the computer on their desks while the resident 
students experienced problems such as, “slow downloads and navigations” when using 
the computers in the different laboratories across campus. The VTE students understood 
their computer because they were the only individuals using the system. On the other 
hand, the resident students could not ascertain what the previous student did to the 
computer before the resident student used it. This has been a problem with the computer 
laboratories on campus -- each laboratory is different and each computer within the 
laboratory is different. This arrangement is not conducive to learning because the 
students divert their attention from the assignment to fixing the computer problem. 

Having more experience in taking web-based courses could be another factor in 
better user satisfaction for the VTE students. By interacting more with the computer than 
the resident students, the VTE students felt more comfortable in handling “technology 
problems” associated with long-distance learning. 


54 



VII. 


CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Based on the results, one can conclude that the questionnaire is an enabling tool 
that can be used to measure the effectiveness of instructional design and usability in web- 
based instruction courses even though the questionnaire was not fully validated by a 
comprehensive study. The trial application was limited to test an initial set of principles 
in both areas of study and to establish a foundation for continued research into measuring 
how usability and instructional design are not independent of each other and why they 
must be taken into consideration when constructing courses for the Web. The data 
collected from the hybrid and VTE sections and the post-questionnaire interview with the 
instructor are consistent with the background information and the initial testing of the 
principles. 

The design for the questionnaire was based on input and feedback from actual 
users who had experience in web-based instruction, human-computer interface, and 
usability engineering. Having subject experts participating in the design assisted in the 
final questionnaire being more effective than the original one constructed for the initial 
testing. Furthermore, user input provided more validity to the findings and helped to 
support the principles for web-based education. The rating scale and anchoring system 
from the QUIS questionnaire was highly instrumental in capturing and categorizing the 
limited data. 

Nevertheless, the questionnaire contained flaws such as baiting statements, which 
assumed that there should have already been something in place. For example, a 
statement such as. Instructional examples were provided insinuates that there should have 
been examples of materials, which is to say that having examples of materials is the only 
correct way to instruct. This is false. As a result, this caused redundancy because 
questions were repeated but with different phrasing that rendered some questions to be 
weak. 
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The major flaw of this study was not receiving data from a purely web-based 
course. The survey attempted to target this type of class. The resident and non-resident 
students had the advantage of the instructor being physically present vice virtually 
present. When there was a problem such as the firewall connectivity, it was remedied 
quickly through the physical interaction with the instructor. A web-based course class 
does not have this advantage. However, the error of not obtaining the data demonstrated 
that communication in web-based education should be simple and direct as possible to 
avoid errors. 

Although there were similarities between our principles and principles from the 
readings, the major difference between previous studies and this study that warrant some 
attention is in the attempt to incorporate human-computer interaction and usability into 
the study. These issues are only mentioned in the readings of instructional design, but 
they do not go into depth. The objective was to see how web design and the usability of a 
system could affect the outcome of instructional design: Does the usability of a system 
affect the educational outcome of the student? In this study, the non-resident students 
showed a higher user satisfaction than the resident students in computer use and 
comprehending subject material. However, the percentages of the grades received 
between the two groups were nearly identical (although grades were not used as a 
criteria). Nonetheless, the data shows that web-based education can succeed through 
good instructional design methods and a usable system. 


B. RECOMMENDATIONS 

Revising and retesting of the questionnaire is necessary in order to improve its 
effectiveness as a measuring tool for rating web-based education courses. One area that 
requires improvement is devising a section for users to rate how user-friendly the 
questionnaire is. Furthermore, weeding out the redundant questions such as questions 72 
and 74 will streamline the questionnaire by not only simplifying the phrasing of 
questions, but also reduce the amount of ledundant questions. The sequencing of the 
questions should be kept the same. 
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Another aspect missing from the questionnaire were questions pertaining 
demographics. Including these questions would assist researchers in assessing and 
identifying different types of groups who take on-line courses. Identifying the different 
traits should provide more thorough research. 

Retesting the questionnaire on purely web-based courses will ultimately prove the 
effectiveness of the questionnaire. Cooperation between NPS and deployed units would 
provide a strong foundation for testing. Deployed ships offering training and classes via 
the Web for the Marines and Sailors, and all oversea bases offering web-based instruction 
courses could benefit from using this questionnaire to assess their on-line courses, while 
demonstrating the effectiveness of the questionnaire as a rating mechanism. 
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APPENDIX A. INITIAL EVALUATION OE PRINCIPLES 


To test the principles and measure the effectiveness of the Instructional Design- 
Usability questionnaire, an initial evaluation was conducted. The project went through a 
full iteration of the design-prototype-analyze cycle while demonstrating an understanding 
of usability engineering, evaluation and design methods, and general knowledge of 
human-computer interaction. 

Six parts were performed: 1. Problem definition 2 Requirements gathering 3. 
Design 4. Prototype 5. Usability Analysis and lastly, 6. Redesign. Each part is defined 
and shows the work performed. 

A. PROJECT ONE: PROBLEM DEFINITION 

This phase consisted of finding and stating a problem in one sentence for clarity. 
An explanation was given for what activities the system supported, who would use it, the 
criteria for judging the system, and how would the problem be approached. 

1. Problem 

The Naval Post Graduate School (NPS) has been actively developing new online 
courses, but lacks an effective tool to measure the quality instruction in its long distance 
learning program. 

2. Proposal 

a. Solution to the Problem 

Using instructional design, human computer interface (HCI), and usability 
principles, a web-based prototype course will be designed that will demonstrate how to 
create an effective online course. 

b. Intended Target Audience 

This courseware is web-based and can be accessed from any location. 
Moreover, the intended target is for instructors and students. The instructors should be 
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able to put a course online with very little difficulty, and students should learn with ease 
from the material. 


c. Criteria to Evaluate the Project 

A formal evaluation will be conducted to ensure the usability of the 
system. The criteria for judging this site will be how well the instructors and students 
navigate the site. The instructors will follow a set of instructional design principles to 
build the course and the students will navigate the course via instructions and take an 
exam to demonstrate their acquired knowledge. The students will then evaluate the 
course with a questionnaire that contains questions regarding how the content was 
presented and the usability of the system. The main consideration for the evaluation is 
ease of putting a course online and the ease of learning online. 

The procedures to evaluate the sessions will be as follows: 

Criteria for Instructors 

• Time to complete the development of a course online. 

• Number of faculty failed to develop a course online. 

• Number of times the help section is used. 

• Number of times user becomes frustrated and dissatisfied with the 
course. 

Criteria for Students 

• Time to complete a course online. 

• Number of students who complete the course exam. 

• Number of times the help section is used. 

• Number of times user becomes frustrated and dissatisfied with the 
course. 
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Based on these criteria and the user questionnaire regarding instructional 
design and usability, the data will be analyzed to correct instructional design and web 
design problems. 

d. Project Design Approach 

The design approach throughout the project will be a customer-oriented for 
students as well as faculty. Since the courseware is already complete, a prototype will be 
built using various tutorials regarding Internet middleware from students at NFS. After 
each design phase, informal user input will be immediately solicited about the design. 
Students from various curricula will evaluate the design by taking the course. 

The conceptual design will be created first followed by the semantic design. The 
physical design will be the third in the sequence of events in developing the course. 
During this phase, a rapid prototyping tool will be used to develop a prototype that gives 
the user the look-and-feel of the actual product. Using crayons and pencils, the design 
will be sketched out for peers to evaluate. After redesigning and revising the design of the 
course, the course will be placed online for users to take. 

B. PROJECT TWO: REQUIREMENTS GATHERING 

A needs analysis, user analysis, and task analysis were devised to understand the 
requirements for the user and the system. A needs analys is establishes the fact that a new 
system is needed due to external demands by determining basic goals, purposes, and 
features for the application. The goal, situation of concern, what activities do the 
designers support, and listing and explaining the features of the system are issues that 
assist in defining what a user will do with the system. 

A user analysis produces a description of the user or user group based on 
significant characteristics and limitations that might affect the design of the interface. 

Information about the work environment, job functions, users’ tasks, and 
characteristics and skills are taken into consideration. Determining how the system will 
be used, how often, and general computer skill level are examples of defining the user 
analysis. 
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A task analysis identifies how users will use the system. A detailed description of 
tasks, subtasks, and methods that the user and system will share in order to complete a set 
of tasks are identified through a top-down decomposition. The purpose of this analysis is 
to avoid duplicating a set of current procedures that the user and computer already 
perform. 

1. Needs Analysis 

a. Goal 

Develop an interactive interface that allows instructors to develop an 
online course and students to learn from the course. 

b. System Assumptions 

• Site is accessible from any location in the world. 

• Software runs on a different server from the NFS main server. 

• Essay questions will not be graded automatically due to various 
subjective answers. 

c. Instructor Assumptions 

1. Interface automatically displays options for construction of course. 

• Buttons for displaying where to place course material. 

• Instructors begin “clicking” buttons to fa mi liarize themselves and feel 
comfortable with the product. 

• Instructors begin placing material on the site. 

2. Instructor does not have any knowledge of developing a course online. 

d. Student Assumptions 

1. The interface automatically displays options for navigating the site. 

• Buttons for navigating the site. 
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• Students begin “clicking” buttons to familiarize themselves and feel 
comfortable with the product. 

• Students begin learning material from the site. 

e. System Features 

1. Provide an easy interface for placing material on online for an 
educational course. 

2. Provide an easy interface for learning material from an online course. 

3. Allow chat rooms for discussion. 

4. Allow email to contact students and instructors. 

5. Allow email link to view additional sites and support course material. 

6. Allow welcome board for placing course materials. 

7. Allow instructor to place buttons that students will need to access 
information for the course. 

8. Allow instructor/student to return to home page when lost/confused. 

9. Allow instructor to view course material after placing material in 
respective places. 

10. Allow instructor to modify material for future courses/exams. 

11. Allow instructor to form various exams such as, multiple choice, 
true/false, essay, fill-in-the-blank. 

12. Allow instructor to place percentage points on questions. 

13. Allow instructor to show incorrect and correct answers. 

14. Allow instructor to place time limits on exam. 

15. Allow instructor to place time limits on each question. 
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2 . 


User Analysis (Situation of Concern) 

a. User Group 

The user group will be instructors and students. These users possess some 
elementary computer experience such as web browsing. They may be novice/casual users 
who will not tolerate learning how to navigate a site while attempting to learn a particular 
subject. Therefore, the design of interface must be so basic that the instructors and 
students can perform their tasks with ease. 

b. Online Community 

This application is designed to create an online community of users as the 
users learn via the web. The users must feel comfortable using this site while they are 
learning online. The ease of this courseware allows students to asynchronously 
communicate with each other through chat rooms, email, discussion board, and the 
welcome site. These features should strengthen the students and instructor into an 
intellectual community, so they can take/support future courses in long distance learning 
via the web. 


c. Simplicity of the Interface 

The users can be expected to have a low frustration level. The interface 
should be kept as simple as possible. All buttons/options should be easily understood. If 
the users spend more time trying to navigate the system, they will not be inclined to learn 
the course material and will not enjoy learning via online. 

3. Task Analysis (Activities Supported) 

a. Instructor Activities 

Primary Task 1: Initiate interface. 

• Enter user name. 

• Enter user password. 

• Go to course panel. 
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Primary Task 2: Understand the make course panel interface. 

• View panel options. 

• Identify which buttons perform what action. 

Primary Task 3: Develop online course. 

• View panel options. 

• Select appropriate option. 

• Load material. 

• Select view material to review material from “student’s point of view.” 

• Return to panel options and repeat steps a-c. 

Primary Task 4: Allow student to enter course site. 

• Enter student user name. 

• Enter student password. 

• Give student email address, username, and password. 

b. Student Activities 

Primary Task 1: Initiate interface. 

• Enter user name. 

• Enter user password. 

• Go to course site. 

Primary Task 2: Understand the course panel interface. 

• View Welcome site. 

• View panel options. 

• Identify which buttons perform what action. 

Primary Task 3: Take online course. 

• Select appropriate button to view course material. 

• Read course material and perform required work. 


65 



Navigate to next eourse folder. 




Primary Task 4: Close interface. 

• Student selects file menu. 

• Student selects Exit. 

C. PROJECT THREE: S\STEM DESIGN 

This portion of the project is the most complex due to unlimited ideas and 
possibilities when discussing design issues. Carroll and Rosson (1985) explain that 
design, as a process is one of the least understood development activities-for user 
interfaces or in any domain. It may be due to creativity and individuality. Requirements 
become lost and neglected for the sake of creativity and individuality. Due to this 
complexity, constant evaluation of design is necessary in order to develop a good 
product. Conceptual design, detailed design, and an early analysis assist in maintaining 
design as a process. Conceptual design is a higher level of design and pertains to 
analyzing objects, attributes of objects, and the relationships between objects, actions on 
objects, actions on attributes, and actions on relationships. 

Detailed design or visual design entails activities such as, wording of messages 
labels and menu choices. How will these object appear on the screen and navigation from 
screen to screen are considerations that designers must take into account. A simple and 
effective method is using paper, pencil, and crayons to sketch the screens that are needed 
for the design. Based on the functionality outlined in the task analysis and the function 
points in the conceptual design, designers should determine how this functionality will be 
addressed via menus, buttons, dialogue boxes and everything else that will be used for the 
actual version of the finished product. 

When conducting an early analysis, designers should show their sketches to as 
many reviewers as time permits who are unfamiliar with the project. The designers 
should walk them through the designs and tasks to obtain some initial feedback as to the 
efficacy of the design. Writing down the comments from the given constructive criticism 
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helps designers to see keep their design according to the specified requirements of the 


user. 


1. Conceptual Design 

NOTE: Designing for this was difficult because the courseware for the online 
course was already created. This was a framework that the researchers could not change. 
Taking this constraint into account, the focus of this phase was on incorporating the 
instructional design and usability principles and how they could be used as a measuring 
tool to evaluate how effective the content of information is for the student. 

a. Objects 

Student Interface Window Buttons 

• Announcements 

• Course and Information 

• Staff Information 

• Course Documents 

• Assignments 

• Communication 

• External Links 

• Student Tools 

• Resources 

• Search 

• Logout 


Instructor Interface Window Buttons 


The instructors Interface will have the same features as the students but 
they will also have a control panel button that allows instructors to build a course. 

b. Object Attributes 

The students and professors will have the option of deciding their type of 
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buttons, colors, and shape of the buttons. 


Student attributes 


• Username 

• Password 

• Registered courses 

• Age 

• First Name 

• Last name 


Faculty Attributes 

• Password 

• Registered courses 

• Age 

• First Name 

• Last name 

• Courses to be taught 


Message Attributes 

• From attribute 

• To attribute 

• Subject attribute 

Course Attributes 

• Course number with departmental code 

• Course title 

• Description of course 

• Goals of course 
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• Prerequisites for course 

• Professor of Course 

Assignment Attributes 

• Course Name 

• Due Date 

• Guidelines 

• Assignment Number 

c. Relationship between Objects 

• The interface window contains the welcome board 

• User begins navigating the course via the options and instructors 
comments. 

• Only instructors have access to the control panel where they will have 
numerous options to build the course. 

• Each student will have his/her username and password to access the 
site. 

• Students and instructors can post messages for assignments. 

• Instructor can post readings, assignments, and exams for students. 

• Students can submit assignments electronically prior to due date. 

• Course will have one or more courses. 

d. Action Object Attributes 

• Users are able to navigate the site. 

• Users are able to change button colors 

• Users are able to change shape of buttons to square or circle 

• Users are able to navigate the site and return to Welcome Page 

• Users are able to enter a chat room 

• Express ideas, thoughts, and problems about assignments 

• Eeave the chat room 
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e. Actions on Relationships 

• Various options can be disabled for the eourse if the instruetor deems 
neeessary. 

/. Project Concessions 

• The ehat rooms and the eourses will not be implemented. 

• Material from one eourse as a design will be shown. 

• Coneentration on the instruetional design prineiples will be an 
emphasis sinee the eourseware has already been ereated. 

2. Visual Design 

Note : the sketehes for this portion will not be ineorporated into the thesis beeause 
the design eould not be transferred eleetronieally. 

After entering the username and password, the announcement page displays any 
news the student needs for the eourse. Direetions for the eourse may also be plaeed in the 
announeement page. The eourse material appears in the eourse information folder. The 
student ean then proeeed to navigate the eourse by ehoosing various options. 

A tutorial appears after the student elieks on the eourse information folder 
allowing the student to read and study the material. 

The large magnitude of symbol seleetion and ieon plaeement possibilities makes 
it diffieult to illustrate these via sketehes. Those developed by the low fidelity tests 
subjeets are ineluded in the Early Analysis. 


3. Early Analysis 

Note: For simplieity, the evaluator’s Task List and one Low Fidelity Test are been 
submitted as examples. 

Four test subjeets were individually asked to evaluate the eontent of material and 
navigability of the eourse. Eaeh subject was asked to read and then explain the eontent of 
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the material. From these explanations and more suggestions from the participants, 
improved principles and a measuring tool to build an effective course were derived. 


Summary of analysis 

• The content was presented in “small understandable paragraphs.” 

• The phrasing and wording of folders such as, guidelines and organizing 
tasks was a big problem. 

• Participants commented regularly on layout, design of product, and its 
usability rather than commenting on the content. 

• The design of the system does not affect how experienced or 
inexperienced users use the system and learn from the material. 

• From eyaluator’s perspective: The instruction design principles need to be 
improved and more emphasis on creating a better checklist for designing a 
course needs to be implemented. 

These results will be considered carefully and incorporated into an improved 
design before proceeding with the prototype implementation. 
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Project 3 Task List 

Evaluator’s copy of the task list 

Reference p. 50 of Class Notes, Develop the Experiment 

Question: How do we dennnstrate that the instructional design principles 
and usability principles enhance weh-based instruction? 

Questionnaire, analysis of errors: 

1. Student understands the Welcome Page instructions. Benchmark: Student does 
not feel “overwhelmed” by the content. (Information is presented in small 
chunks). 

a. Read instructions 

b. Explain instructions. 

2. Student is comfortable with course instructions. Benchmark: Student navigates to 
Course Documents. Student does not feel “overloaded” by the content. 

a. Read course objectives 

b. Read syllabus (class schedule) 

c. Read negotiating guidelines 

d. Explain a - c. 

3. Student comprehends presented material. Benchmark: Student reads material, 
takes a quiz, and passes quiz. Student does not feel “overloaded” by the content. 

a. Read material. 

b. Explain material 
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Project 3: Example of Low Fidelity Test Evaluation 

Tester: Erich Stefanyshyn 
Subject: Male, USMC, Captain, 31 years 


Overview: The evaluation lasted 25 minutes. During this time, the subject was 
given various outlays of the interface and asked to review the content of material on the 
“screen.” 

Subject’s first impressions: 

Task 1: Student did not feel “overwhelmed” by the material. He thought that it 
was organized and could understand the intent of the course. However, He wanted to see 
the message once. He thought that a “E' time for visitors” message should be placed to 
avoid redundancy and boggle of material. 

Task 2: Student did not feel “overloaded” by the material. However, the content 
could be better explained. For example, when meeting, the instructor needs to explain 
that the meeting will be “meeting online asynchronously” vice “meeting 
asynchronously.” Another good recommendation is meeting synchronously “via chat 
room and discussion boards at your (groups convenience).” Furthermore, there should be 
some directions on where to navigate next from the course objectives folder to the class 
schedule folders. It helps to “channel the transition from folder to folder.” 

Task 3: F‘ Part: The material in the explanation folder was not overwhelming, but 
it was not organized as well as the previous Task 1 and 2. For example, there should be 
check boxes assigned to the instructions to remind users (inexperienced and experienced) 
where to go and what to do. When they finish, they can check the boxes. The objectives 
and assignments should be simpler. Also, place time limits on the reading assignments 
and keep a, “instant review sheet next to the assignment for venting purposes.” For 
example, if an assignment lasts one hour and the student takes 2 hours to complete the 
reading, the student should go back to his checklist and be able to let the instructor know 
that there the assignment was “overload.” Place a message on top for students to critique 
honestly to let them know that this is their site and education. 
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2“*^ Part: This should be the “last hyperlink because it takes the most time to do.” 
Content is sufficient, but there is too many paragraphs. Incorporate more “bullets” for 
both users. For “inexperienced, to see ideas and experienced to go directly to the 
material.” Also “hyperlink the bullets for the paragraphs”. Overall, the buttons for 
navigating are confusing. For instance. Assignment button = test and should be a daily 
task like course documents. Also, divide the lessons into day/time periods. For example, 
Weekl: day 1, 1 hour limit, day 2, 1 hour limit, etc. 

Subject’s Opinion: Subject noted the need to change some of the buttons for 
better clarification. Subject stressed that the material needs to be as clear and concise as 
possible to avoid confusing the student. Subject would also like to see the live program. 

D. PROJECT FOUR: PROTOTYPE 

Based on the results from the design in Phase 3, a prototype is built for more 
testing and evaluation. It involves production of at least one early version of the system 
that displays the important features of the system in operation. Prototyping is very 
beneficial to a project because it involves user participation and lets developers observe 
how the users interact with the system. In short, like a car, the user is “taking the system 
for a spin” (Hix & Hartson, 1993). 

A prototype for the web-based evaluation was constructed using the school’s 
courseware. A course based on Internet Middleware was put online. The lesson involved 
Web/Site Management Tools for constructing Web pages. It was built according to the 
measuring tool that was made following in Project 3 and issued in project 5. 

E. PROJECT FIVE: USABILITY ANALYSIS 

A usability analysis allows users and developers to see how well the design and 
the prototype were constructed, and measure its effectiveness and usability. 

Planning well for a usability analysis is imperative for its success. The testers 
should ensure that a set of tasks have been planned for the users. The tasks should 
measure what the system is supposed to be used for. Users who will interact with the 


74 



system should be chosen as subjects. The more subjects tested, the more feedback can be 
garnished for analysis to assess the usability of the system. 

At this stage, data collection is more formal and contains both qualitative and 
quantitative elements. The same questions asked in the phase three are asked more in 
depth. Evaluators are looking for where the system did not meet user expectations. 
Efficiency is one method of evaluation. How optimal did the users perform the tasks? 
How long did it take to complete the tasks? Could these tasks been made any easier? 
Were there delays in the system (where the user spent time thinking about what to do 
next)? How can these be removed or minimized? 

Errors are also analyzed. Do users make errors? If so, can they be engineered out 
of the system? If not, has the design been adequately accounted for them by being 
recoverable and informative messages placed to assist the user? What errors were made 
and were they critical? 

Learnability/Memorability is taken into account by evaluating how easy the 
system was easy to use. Will users need help and training? Most importantly, how well 
does the error rate drop as the user become more familiar with the system? If not, the 
problems need to be identified and corrected. 

Finally, user satisfaction with the system is also evaluated. Would they like to see 
it on-line for real usage? What changes did they suggest to make? Usually the user’s 
suggestions bring out ideas that were not thought about previously. Finally, the 
measurement of user satisfaction is when the user does comment negatively about the 
system and does not say anything while using the system. This demonstrates that the user 
is heavily engaged and the system is easy to use. 

1. Results from the Usability Analysis 

The Test Tasks outlined in Appendices 1 and 2 were designed to observe and 
record how the users interacted with the presented material. Some of the interactions 
involved mouse and button manipulation based on instructions within the content. 
Appendix 3 demonstrates the post course questionnaire that the users evaluated after 


75 



taking the course. When users did not know what to mark, they were referred to the Task 
List assist them. For instance, the first checklist, “Are the students welcomed properly,” 
one subject did not know what to put, and when shown where the welcome phrase was 
located, he was ready to mark 5. But he was told to judge the guideline when he was at 
that page during the course session. He changed his opinion to 3 because “he did see it 
but he did not feel as welcomed because it did not stand out and grab him.” Finally, at 
the beginning of the each interview, the importance of comments and criticism of the 
interviewee was highly expressed and encouraged. Throughout the test and interview, the 
evaluators constantly encouraged the users to comment and think loudly. 

Also contained, as appendices are the consent, minimal risk, privacy act 
statement, and participant demographics forms. 


a. Efficiency 

Were the users able to navigate based on the instructor’s comments? 

The subjects could navigate on the instructor’s comments. After reading 
the instructions, the students spent very little time to find the buttons and utilize the 
system. A typical question during the process was, “Is this the button to press?” or “Do I 
go here?” After clicking on the button(s), the subjects immediately understood how the 
system worked and could concentrate on the material. 

To better the course, all subjects preferred that a hyperlink be placed next 
to the final instruction instead of having to search for the button. Otherwise, the content 
was straightforward. 


Did the computer interface interrupt the course content? 

As long as they could follow the course, the subjects felt that the system 
was easy to use, and that they could concentrate on the material. Since none of the 
subjects used Help, we saw that as a good indication that they were concentrating on the 
material vice fighting the system. 
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Did the users feel that any tasks could have been easier to perform? 


During the presentation of the material, as the subjects read and clicked on 
various web sites, they would have liked to have seen the reading objectives, 
assignments, and review questions on the reading pages in addition to being in the 
Course Syllabus. 

The additional links within the material should be placed after the 
paragraph because the readers preferred to read the material first and then have the 
“option of clicking the link for more information.” One subject wondered about the use 
of links when she said, “this could go on forever, when do I stop?” Placing the links after 
the paragraph would structure the material better and let the students read the main 
content and then assists students who are interested to visit other sites. The uninterested 
students could continue to the next paragraph of information. One subject, for example, 
did not click on any hyperlinks because he did not know where he would go and also did 
not know if he should have clicked on them. 

Overall, the learning material was overwhelming due to the lack of 
structure. For instance, “The web sites had a lot of information, and at what point do I 
stop [navigating]?” The content should be organized better and an explanation the 
websites should be provided to the students. This way, students will know what to expect 
from the site and when they should stop. 

b. Errors 

Could context errors be engineered out of the system? 

All subjects complained about word misspellings. “It distracts me from the 
reading. Now I have to start all over again. I don’t like typos in my textbooks nor do I 
like them on my computer.” 

Bolding was another issue the subjects mentioned when they navigated the 
site. The wording in each folder was identical. Nothing “stuck out and grabbed my 
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attention.” At the Welcome Page, although there was a “welcome comment,” it was hard 
to distinguish because it was in plain text and meshed in with the other material. As a 
result, two of the subjects “did not feel welcomed.” 

The biggest complaint involving errors was the lack of indentations in all 
folders (especially the Course Guidelines folder). We failed to indent the material and it 
made the content difficult to read and understand. 

Did the system account for errors ? 

No, this where the system falls short of expectations and the instructor 
should compensate for them. The system itself did not allow for indentation, spelling 
check, and bolding. When we tabbed, the cursor went to another location on the screen 
Even though we could have used the space bar to compensate, we did not because it was 
time consuming. Nevertheless, by not investing the extra time, and effort caused 
problems for our users. This lack of system functionality causes hardship on the 
instructor (spending time) and students (dealing with the problems). 


What critical incidents could be identified? 

One potential (and greatest) error associated with the course is content 
instructions and linking to other websites. If the instructions are not clear and the web 
sites are not explained, students could spend endless hours and wonder when to stop. 
They could become frustrated with the system and the course, and eventually, lose 
interest with the online education. “The web site should be used to explain and enhance 
the material.” This avoids information overload and makes learning easy. 

c. Learnability/Memorability 

Was the system easy to learn ? 

The system was easy to learn. The users did not have complaints about the 
product. The wording of the button features was different than what they expected. Even 


78 



though there was not an exam, the users associated the Assessment folder for assignments 
rather than for exams. 


Did the system confuse the student? 

Like easy to learn, the system did not confuse the subjects. In the 
beginning of the course, they adjusted to the interface by clicking on various buttons. 
Thereafter, they concentrated on the material. 


Did the content confuse the student? 

The students were confused in the Course Guidelines, Syllabus, and 
Course Documents folders. The confusion was due to the lack of organization and the 
already mentioned errors. The course subject was pleasing, but the content needed to be 
structured better. 

d. Satisfaction 

Were the users satisfied with the system? 

The users enjoyed the system. They did not go to Help for assistance. 
When they became misguided, they clicked back until they found a page they were most 
comfortable and familiar with and restarted. 


Were the users required to use “Help?” 
The users did not use this feature. 


Were the users satisfied with the course content? 

Overall, the users did like the content. Some of the subjects were already 


familiar with the information and skipped to other topics. They also enjoyed surfing the 
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websites (Web Monkey ms the most used). As for the material being presented in small 
chunks, most of the users felt that the bits of information was sufficient. However, there 
should be a main textbook for the course. Furthermore, all subjects noted that the material 
was for students who are interested in Internet middleware and/or for higher-level 
students. Finally, a glossary page should be added or terms should be listed with the 
assignments so the student does not feel “belittled.” 
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Would they like to see it online for real usage? 

The subjects did enjoy the course and believe that they could learn from it 
if it is placed online as an actual course. They would like to see the changes. 


2. Analysis 

For web-based instruction, system usability is dependent on good instructional 
design as instructional design is dependent on a good reliable system. Even though the 
computer system is running perfectly, the user may become dissatisfied with an online 
course because the content is not organized and coherent. If students spend more time 
wrestling with the presentation of content rather than the content itself, they begin to 
divert their time from the content to adjust to the presentation. As a result, students spend 
more than the usual amount of required time on a course and may become frustrated with 
long distance education. This defeats the purpose of long distance education that allows 
students to take the course asynchronously for a short amount of time. 

The instructors are another aspect to consider because they have to put the course 
online. If the instructors do not have the proper guidance and tools to transfer a traditional 
course into a web-based course, they also become frustrated with the online education 
process, because they have invested time into their courses to be successful. The 
courseware may function properly, but if they do not know how to put the course online, 
then it is better to have no online course than a bad online course. 

Instructors also need to compensate for a product’s weakness through 
instructional design. Although the intent of this research is not to judge the product used, 
but show that constraints of systems do affect how a course is put online. Sometimes the 
constraints affect the intended user, which in this case, is the student. Instructors need to 
put on their creative hats to compensate for the constraints by adding audio and/or text 
related photos. By the same token, by placing the course contents on file and having 
students link to the file from the courseware site may compensate for the lack of bold, 
indents, and reduce typos. However, the increased linking can also hinder learning due to 
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more navigating which increases the chance of becoming “lost”, and the file document 
may not be presented as well via the system. 

The guideline checklist reeds to be reworded so that the instructor or students do 
not just “check the block.” For instance, guideline 11 says, a syllabus (class schedule) is 
in place. It should also be followed by how clear is the syllabus and is it presented in 
small chunks! After all, the intent of the checklist is to assist instructors to put an 
effective course online, and for students to recognize what basic ingredients are necessary 
to constitute a good online course. A heuristics evaluation using better benchmarks to 
measure the users’ behaviors. 

The course content needs to be revised. More indentation, bold, and hyperlinks 
should be placed in better strategic places. 

Finally, the broad range of participants from various curricula provided a very 
good insight of how people view the course. The student from the Information 
Technology department was more interested in the buttons whereas the student from 
System Management cared more about presentation (fonts, indents). For future 
considerations, more testing involving various backgrounds of students to determine how 
effective a course could be tailored according to an improved checklist. 
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Project 5 Evaluator’s Task List 

Reference p. 50 of Class Notes, Develop the Experiment 


Question: How do we demonstrate that the Instructional Design Principles 
enhance web-based instruction? Does the usability (computer interface) 
affect/distract the user from the content? How effective is the questionnaire as a 
measuring tool for web-based instruction? 

Note: Due to time constraints, we will not test benchmark four. It will remain in 
the benchmarks for future consideration. 

Benchmark 1: Does the student understand the instructions on the Welcome 
Page? Student does not feel “overwhelmed” by the content. (Information is presented in 
small chunks). 

Ask the subject to read and explain the content. 

Benchmark 2: Is the student comfortable with course instructions? Student 
navigates to Course Information Documents. Student does not feel “overloaded” by the 
content. 

a. Read course objectives 

b. Read syllabus (class schedule) 

c. Read course guidelines 

d. Explain a - c. 

Benchmark 3: Is the student comfortable with course instructions? 

Student navigates to Course Documents. Student does not feel “overloaded” by 
the content. 

a. Read course objectives 

b. Explain material 

Benchmark 4: Does the student comprehend presented material? 

Student reads material, takes a quiz, and passes quiz. Student does not feel 
“overloaded” by the content. 

a. Read material. 

b. Explain material 


83 



Project 5 Participant’s Task List 
Reference p. 50 of Class Notes, Develop the Experiment 


You are testing the course material. The expected time is 30 minutes. During this 
time, you may read and think aloud. When you are finished with the mini-course, we will 
ask questions pertaining to the material. This part will last no longer than 20 minutes. 
Whenever the interface distracts you from the content, please say what feature distracts 
you and why. 


Instructions for the course 

1. Read the material on the “Welcome” Page. 

2. Read the material on the “Course Information” Page. 

3. Read the material in the “Course Documents” page. 

4. Explain all sections. 

5. Using the questionnaire, rate the sections and the material. 
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Project 5 Consent Form 


1. Introduction. You are invited to participate in a study of web-based instruction. With 
information gathered from you and other participants, we hope to discover insight on how 
Instmctional Design principles and Human Computer Interface/Usability principles affect 
online learning. We ask you to read and sign this form indicating that you agree to be in the 
study. Please ask any questions you may have before signing. 

2. Background Information. The Naval Postgraduate School Distance learning is conducting 
this study. 

3. Procedures. If you agree to participate in this study, the researcher will explain the tasks in 
detail. There will be two sessions: a) 30 minute mini-course at a web site and 2) a post 
interview discussing the content of the course and any distractions the system caused to the 
subject. 

4. Risks and Benefits. This research involves no risks or discomforts. The benefits to the 
participants are understanding web-based instmction and contributing to current research in 
web-based instmction. 

5. Compensation. No tangible reward will be given. A copy of the results will be available to 
you at the conclusion of the experiment. 

6. Confidentiality. The records of this study will be kept confidential. No information will be 
publicly accessible which will possibly identify you as a participant. 

7. Voluntary Nature of the Study. If you agree to participate, you are free to withdraw from the 
study at any time without prejudice. You will be provided a copy of this form for your records. 

8. Points of Contact. If you have any further questions or comments after the completion of the 
study, you may contact the research supervisors, Capt Erich I Stefanyshyn or Lt Mehmet 
Sezgin at (408) 656 - 4071 (Email: eistefan@ .nps.navy.mil or msezgin@nps.navy.mil). 

9. Statement of Consent. I have read the above information. I have asked all question and have 
had my questions answered. I agree to participate in this study. 


Participant’s Signature 

Date 

Researcher’s Signature 

Date 

Researcher’s Signature 

Date 
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Project 5 MINIMAL RISK CONSENT STATEMENT 
NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 


Subj: VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN: WEB-BASED 

INSTRUCTION EXPERIMENT 

1. I have read, understand and been provided “Information for Participants” that provides the 
details of the below acknowledgements. 

2. I understand that this project involves research. An explanation of the purposes of the 
research, a description of procedures to be used, identification of experimental procedures, 
and the extended duration of my participation have been provided to me. 

3. I understand that this project does not involve more than minimal risk. I have been informed 
of any reasonably foreseeable risks or discomforts to me. 

4. I have been i nf ormed of any benefits to me or to others that may reasonably be expected from 
the research. 

5. I have signed a statement describing the extent to which confidentiality of records identifying 
me will be maintained. 

6. I have been informed that since the risks are minimal any injury I suffer while participating in 
the experiment will be at my own risk and that I accept full responsibility for my own 
medical treatment. 

7. I understand that my participation in this project is voluntary. Refusal to participate will 
involve no penalty or loss of benefits to which I am otherwise entitled. I also understand that 
I may discontinue participation at any time without penalty or loss of benefits to which I am 
otherwise entitled. 

8. I understand that the individual to contact should I need answers to pertinent questions about 
the research are Rudy Darken, Ph.D. and Anthony Ciavarelli, Ph.D., Principal Investigators, 
and about my rights as a research subject or concerning a research related injury. A full and 
responsive discussion of the elements of this project and my consent has taken place. 


Signature of Principal Investigator(s) 

Date 

Signature of Volunteer 

Date 

Signature of Witness 

Date 
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Project 5 PRIVACY ACT STATEMENT 


1. Authority: Naval Post Graduate School, Monterey, CA 93940, Naval Instruction 

2. Purpose: Web-based instruction, instructional design principles and Human 
Computer Interface (HCI)/Usability principles will be studied to enhance 
knowledge, and/or improve long-distance education programs at the Naval 
Postgraduate School. 

3. Use: Information on web-based instruction will be used for analysis by the 
Departments of the Navy and Defense, and other U.S. Government agencies, 
provided this use is compatible with the purpose for which the information was 
collected. Use of the information may be granted to legitimate non-government 
agencies or individuals by the Naval Postgraduate School in accordance with the 
provisions of the Freedom of Information Act. 

4. Disclosure/Confidentiality: 

a. I have been assured that my privacy will be safeguarded. I will be 
assigned a control or code number, which thereafter will be the only 
identifying entry on any of the research records. The Principal 
Investigator will maintain the cross-reference between name and control 
number. It will be decoded only when beneficial to me or if some 
circumstances, which are not apparent at this time, would make it clear 
that decoding would enhance the value of the research data. In all cases, 
the provisions of the Privacy Act Statement will be honored. 

b. I understand that a record of the information contained in this Consent 
Statement or derived from the experiment described herein will be 
retained permanently at the Naval Postgraduate School or by higher 
authority. I voluntarily agree to its disclosure to agencies or individuals 
indicated in paragraph 3 and I have been informed that failure to agree to 
such disclosure may negate the purpose for which the experiment was 
conducted. 

c. I also understand that disclosure of the requested information, including 
my Social Security Number, is voluntary. 


Signature of Volunteer, Print Name, Grade/Rank, DOB, SSN, Date 


Signature of Witness 


Date 
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Project 5 PARTICIPANT DEMOGRAPHICS 


Participant ID (assigned by tester): 
Circle the best choice. 

Sex: M F 


Age: under 18 

20-25 

25-30 

30-35 

over 30 

Education level: 

HS diploma 

GED 

BS/BA 

MS PhD 

Computer Experience: 

None 


Novice 

Intermediate Advanced 
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APPENDIX B: INSTRUCTIONAL DESIGN-USABILITY 

QUESTIONNAIRE 


PART 1: Getting the Learner Started 

This section provides a ''big picture view" of the course. It assists students by providing a 
course description, prerequisites for the course, course objectives and goals, navigation of 
the web site, and instructor biography, a course syllabus (of chapters to read, homework 
assignments, tests, etc), and course expectations (of the instructor and student). 

Before continuing with the course, students should be able to understand course 
objectives, syllabus, and expectations, and how to access the course lessons. 

INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 


Not Applicable = NA. 

Instructional Quality 


1.1 

Course Objectives 





1.1.1 Course goals defined 

confusing 

clear 




12 34 

5 67 8 9 

NA 


1.1.2 Brief course description 

confusing 

clear 




12 34 

5 67 8 9 

NA 

1.1.3 

Course prerequisites 

confusing 

clear 




12 34 

5 67 8 9 

NA 

1.1.4 

Students given instructions/orientation 
on course navigation 

unhelpful 

helpful 




1234 

5 67 8 9 

NA 

1.1.5 

Biography of instructor 

unhelpful 

helpful 




12 34 

5 67 8 9 

NA 


1.2 Course Svllabus/Class Schedule 

1.2.1 Class schedule (when class meets) confusing clear 

123456789 NA 

1.2.2 Class agenda (amount of reading per week, 
required assignments due, exam dates) 

confusing clear 

123456789 NA 
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1.3 


Course Expectations 


1.3.1 

Role of the instructor was defined 

confusing 

clear 




123456789 

NA 

1.3.2 

Students’ responsibilities 

confusing 

clear 




123456789 

NA 

1.3.3 

Grading criteria and procedure 

confusing 

clear 




123456789 

NA 


Web Design 

1.4 Course Objectives 

1.4.1 Content from course objectives 
(i.e. font, bold, italics, underlines, 

misspellings, etc) hard to read easy to read 

123456789 NA 

1.4.2 Content provided in understandable 

paragraphs inadequate adequate 

123456789 NA 

1.4.3 Content is meaningfully labeled illogical logical 

123456789 NA 

1.5 Course Svllabus/Class Schedule 

1.5.1 Content (i.e. font, bold, italics, 

underlines, misspellings, etc) hard to read easy to read 

123456789 NA 

1.5.2 Content provided in understandable 

paragraphs inadequate adequate 

123456789 NA 

1.5.3 Content is meaningfully labeled illogical logical 

123456789 NA 

1.6 Course Expectations 

1.6.1 Content (i.e. font, bold, italics, 

underlines, misspellings, etc) hard to read easy to read 

123456789 NA 
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1.6.2 Content provided in understandable paragraphs 


1.6.3 Content is meaningfully labeled 


Please write your comments about the Getting 


inadequate adequate 

123456789 NA 

illogical logical 

123456789 NA 

the Learner Started section here: 


PART 2: Presenting the Subject Material 

This section evaluates the actual lesson/module. This is where the actual learning begins. 
The instructor may have organized the material into weekly folders, topic folders, and/or 
a hyperlink for the week for the student to view or download. 

In this area, the number of options for delivering a lesson ranges from purchasing a 
textbook for the course, additional reading(s) via the web, problems, exercises, 
discussions via chat rooms and/or discussion boards, written assignments (summaries), 
and/or extra web-links to enhance the lesson(s). 

INSTRUCTIONS 


Please circle the numbers, which most appropriately reflect your impressions about this 
course. 

Not Applicable = NA. 

Instructional Quality 


2.1 Purpose of the Course 



2.1.1 

The purpose of each module/section was 
adequately defined 

never always 

123456789 

NA 


2.1.2 

Learning objectives for each component of 
the course were adequately defined 

never always 

123456789 

NA 

2.2 

2.1.3 Weekly/module folders are in logical order 

Instructor’s Particination 

never always 

123456789 

NA 


2.2.1 

Students were encouraged to collaborate 

never always 

123456789 

NA 
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2.2.2 

Instructor gave learning guidance 

never always 




123456789 

NA 

2.2.3 

Learning guidance was adequate 

never always 




123456789 

NA 

2.3.4 

The instructor responded to students’ 
questions/concerns regarding material 

never always 




123456789 

NA 

2.3.5 

Instructor’s participation met student 
expectations 

never always 




123456789 

NA 

Presentation of Subject Material 



2.3.1 

Instructional material was presented at 
proper level 

difficult easy 




123456789 

NA 

2.3.2 

Instructional explanations of material 

confusing clear 




123456789 

NA 

2.3.3 

Instructional explanations included 
useful illustrative examples (pictures, 
graphs, diagrams, etc) 

never always 




123456789 

NA 

2.3.4 

Assigned reading related to course objectives 

never always 




123456789 

NA 

2.3.5 

Instructional content relevant to learning 
objectives 

never always 




123456789 

NA 

2.3.6 

Instructional content in correct sequence 

never always 




123456789 

NA 

2.3.7 

Supplementary web sites provided to 
enhance learning 

never always 




123456789 

NA 

2.3.8 

Subject material was highly interactive 

never always 




123456789 

NA 


2.3.8.1 Interaction of subject material useful 

never always 




123456789 

NA 

2.3.9 

Content engaged the student in active 
learning tasks 

never always 




123456789 

NA 

Assignments for Subject Material 



2.4.1 

Assignment helped achieve 
learning objectives 

never always 




123456789 

NA 
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2.4.2 Review of material was provided 

2.4.3 Instructional practice examples were provided 

2.4.4 Discussion board was used during the course 

2.4.4.1 Discussion board useful 

2.4.5 Chat rooms were used during the course 

2.4.4.1 Chat rooms useful 

2.4.4 Written assignments (term papers, one-page 
summaries, etc) were included in each 
assignment 

2.5 Feedback Section for Students 

2.5.1 Feedback section on lesson available 
for students 

2.5.2 Feedback section on lesson useful 

Web Design 


never always 

123456789 

never always 

123456789 

never always 

123456789 

never always 

123456789 

never always 

123456789 

never always 

123456789 


never always 

123456789 


never always 

123456789 


never always 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 


2.6 Subject material content 

(i.e. font, bold, italics, underlines, misspellings, etc) hard to read easy to read 

123456789 NA 

2.7 Subject material content provided in 

understandable paragraphs inadequate adequate 

123456789 NA 

2.8 Content is meaningfully labeled illogical logical 

123456789 NA 

Please write your comments about the Presentation of the Subject Material here: 
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PART 3: Assessing the Subject Material 

This section rates the exams, projects, term papers, etc. of the subject material. 

INSTRUCTIONS 


Please circle the numbers, which most appropriately reflect your impressions about this 
course. 

Not Applicable = NA. 

Instructional Quality 


3.1 Exam questions relevant to learning objectives 


3.2 Exam questions presented at proper level 


3.3 Exam questions were understandable 


3.4 Exam had a reasonable time limit 


3.5 Exam provided feedback and review 
when questions answered incorrectly 

3.6 Instructional content provided adequate 
exam preparation 

3.7 Student critique section of exam provided 


3.8 Assessments other than exams (papers, projects, 
assignments, etc) were used in this course 


3.8.1 Assessments relevant to learning objectives 


never always 

123456789 

never always 

123456789 

never always 

123456789 

never always 

123456789 


never always 

123456789 


never always 

123456789 

never always 

123456789 


never always 

123456789 


never always 

123456789 


NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

NA 


Web Design 

3.9 Exam content 

(i.e. font, bold, italics, underlines, misspellings, etc) hard to read easy to read 

123456789 NA 

3.10 Exam content provided in 

understandable paragraphs inadequate adequate 

123456789 NA 
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3.11 Content is meaningfully labeled illogical logical 

123456789 NA 

Please write your comments about the Assessing of Subject Material section here: 


PART 4: Assessing the Usability of the Courseware 

Courseware is the software program that is used to deliver the course (Blackboard, Cnet, 
Courseware, etc) online. If the courseware does not function properly (interface, buttons, 
link time, navigation, runs slow), it may be difficult for the student to concentrate on the 
material. 

Input from students on how the courseware functions, is vital for the success of web- 
based instruction because if students cannot use it, then the courseware hindered the 
learning process. 

INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 


Not Applicable = NA. 


4.1 

Student critique provided for courseware 

never always 

123456789 

NA 

4.2 

Courseware was easy to learn 

never always 

123456789 

NA 

4.3 

Courseware was easy to use (after learning it, did you 
struggle with it for the remainder of the course) 

never always 

123456789 

NA 

4.4 

Course site was accessible (connectivity, firewall) 

never always 

123456789 

NA 

4.5 

Course site required to much navigation to 
view actual content (clicking until site reached) 

never always 

123456789 

NA 

4.6 

Were there difficulties in navigating from page to page 

never always 

123456789 

NA 
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4.7 

Were there difficulties in navigating between lessons 

never always 




123456789 

NA 

4.8 

Were there difficulties in navigating from course page 
to web links and back to course page 

never always 




123456789 

NA 

4.9 

Were there difficulties in paging from top of page 
to bottom of page 

never always 




123456789 

NA 

4.10 

Were there difficulties in paging from bottom of page 
to top of page 

never always 




123456789 

NA 

4.11 

Were there difficulties in finding the 
“home” page 

never always 




123456789 

NA 

4.12 

Were there difficulties in finding the 
desired web page (location) 

never always 




123456789 

NA 

4.13 

Speed of Courseware was adequate (waiting a long 
time to go to next site and for downloads) 

never always 




123456789 

NA 


Please write your comments about Assessing the Usability of the Courseware here: 
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APPENDIX C: QUESTIONNAIRE EOR USER INTERACTION 

SATISEACTION 


The following scanned document is a more detailed explanation of the The 
Questionnaire for User Interaction Satisfaction (QUIS) 7.0 developed by the University 
of Maryland to assess the usability of computers/systems as explained in Chapter IV. The 
explanation begins on the following page due to the format of the scan. 

The University of Maryland granted permission to reprint this document. For 
more information regarding QUIS 7.0 can be obtained by visiting 
www.otc.umd.edu/Gatewav/Summer98/transfer.html or writing the Office of Technology 
Liaison at otl.umail.umd.edu. 
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About the QUIS™, version 7.0 


The Questionnaire for User Interaction Satisfaction (QUIS™) is a tool developed by a multi-disciplinary 
team of researchers in the Human-Computer Interaction Lab (HCIL) at the University of Maryland at 
College Park. The QUIS™ was designed to assess users’ subjective satisfaction with specific aspects of the 
human-computer interface. The QUIS'''” team successfully addressed the reliability and validity problems 
found in other satisfaction measures, creating a measure that is highly reli^le across many types of 
interfeces. 

QUIS''” 7.0 is the first update of the QUIS™ content since 1989. it contains a demographic questionnaire, 
a measure of overall system satisfaction along six scales, and hierarchically organized measures of eleven 
specific interface factors (screen factors, terminology and system feedback, learning factors, system 
capabilities, technical manuals, on-line tutorials, multimedia, teleconferencing, and software installation). 
Each area measures the users' overall satisfaction with that facet of the interface, as well as the factors that 
make up that facet, on a 9-point scale. The questionnaire is configurable according to the needs of each 
interface analysis by including only the sections that are of interest to the user. 


PART 1: System Experience 

This section measures the user’s experience with the system or systems being tested. Both length of 
experience and frequency of use are measured. 

PART 2: Past Experience 

This section gauges the user’s experiences with computer systems in general. 


PART 3: Overall User Reactions 

Satisfaction along six high level interface factors is measured in this section. 

PART 4: Screen 

This section measures satisfaction with a number of factors related to visual displays, including fonts, 
highlights, layouts and navigation. 

PART 5: Terminology and System Information 

This section measures satisfaction with system messages, user feedback, and task related wording that the 
system generates. 

PART 6: Learning 

This section measures the user’s perception of their ability to learn complex system tasks. 

PART 7: System Capabilities 

This section measures satisfaction with the system's performance and reliability, both in error recovery and 
error prevention. 

PART 8: Technical Manuals & On-line Help 

This section gauges the system's help facilities to determine whether the information presented is of the 
proper language, scope, and granularity to solve the problems a user is likely to have. 

PART 9: On-line Tutorials 

This section asks users to rate a number of factors that predict the usefulness of on-line tutorials. Factors 
include structure, ease of navigation, level of detail, quality of example tasks, and overall effectiveness of the 
tutorial. 
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PART 10: Multimedia 

This section measures the subjective perception of images, sound, video, and color display within the 
system across several factors. 

PART 11: Teleconferencing 

The ease or difficulty of system installation and configuration, the fidelity of the communication, and the 
ability to share a common experience through distance communication determines the success of a 
teleconferencing experience. This section gauges the user’s subjective satisfaction with these aspects of 
telecommunication. 

PART 12: Software Installation 

The speed, difficulty, flexibility, and level of feedback provided during the installation of a system are 
measured in this section. 

How to Use the QUIS™: Some Questions and Answers 

Can we omit questions to save time or limit scope? 

The hierarchical structure of this questionnaire allows you to be selective in both the coverage and 
granularity of your inquiry. For this reason, the questionnaire can be as long or as short as your situation 
requires. There are large sections of the questionnaire that you may have no need to use. Other sections 
will only need to include the main questions without the sub-questions. In areas that you are particularly 
interested in, you may choose to include the sub-questions also. 

in any case, we recommend that you include the overall measures, as they provide a useful and concise 
picture of the general usability of your system. 

Can we add questions that address our specific needs? 

We generally expect that any questions added that are in the "QUIS™ style" will have a reasonable amount 
of reliability. It is common practice to tune the QUIS™ in this way. 

How might the QUIS™ data be analyzed? 

Several approaches have been'taken in the analysis of QUIS™ data. 

Forming Data Sets: 

Once the QUIS™™ data has been collected, the first decision is whether to analyze the data as a whole or 
in groups. Obviously, if you are comparing two different pieces of software you will want to group the data 
by. say. Software Package A and Software Package B. On the other hand, if you are interested in differences 
between types of users of the same package, you would group on groups of individuals, say. User Group 1 
and User Group 2. When forming data sets, we will be interested in statistics within each group as well as 
hypotheses tests between groups. Let’s first consider the statistics within groups. 

Profiles and Diagnostic Tests: 

One of the most useful analyses, particularly for iterative testing and design, is the profile. The profile 
reveals the strengths and weaknesses of the software program or workstation by showing the deviations of 
the means above and below a criterion. A profile is generated by calculating the means and standard 
deviations for each item in the QUIS™. The means are then graphed on a scale from 1 to 9 as shown in 
the figure below; 

The midpoint of the rating scale (5) can be used as a criterion. If the item is above 5, it is perceived as 
being better than an arbitrary-, mediocre value. However, that is generally not good enough. We mav also 
use the overall mean of the group as a criterion. Such a mean is shown in the figure. 
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Figure 0-1 

It is useful to plot a confidence interval around each mean in order to determine its reliability. The 
confidence interval also indicates whether the mean of an item is significantly above or below some 
criterion. For example, if a 95% confidence interval includes ^ within its boundaries, then it indicates that 
the mean is not significantly different from 5 at the .05 level of significance. 

The profile can be used to identify the areas in the application which are panicuiarly good or particularly 
bad. Start with the item having the lowest mean. Identify flaws in the software that may have led to this 
low mean. Then go to the next lowest item and repeat. Do this until you are satisfied that you have 
identified the major problems. Then start with the item having the hipest mean. Ask yourself why this 
aspect was rated so high and how it can be used to further enhance the software. Then go the next highest 
item and repeat. Again, do this until you are satisfied that you have identified all of the strong points of 
the software. 

A more sensitive and statistically powerful technique for identifying the strengths and weaknesses is to use 
a within-subject approach. For each respondent, compute a mean of all of his or her ratings. Then for each 
of the respondent’s ratings get the deviation between that rating and the respondent’s mean, (dji^ = Xj|^ - 
M.k, where is the rating for item k by respondent i and Mj is the mean for respondent i across all 
Items). A sirnple t-test on these deviations to see if they are significantly different from zero for a particular 
item will indicate whether the item is perceived as high or low relative to each respondent’s average rating. 
In general, it is nice to have a sample size of at least 20 for statistical purposes. However, realizing that 
many usability professionals limit their samples to about 10, it is suggested that one avoid statistical tests 
and generalizations by presenting only means and focusing on the highest and lowest ratings. 

Comparing Groups: 

When your data is composed of groups, you make comparisons between groups at the overall level and at 
the level of individual items. However, remember that the more statistical tests you run, the greater your 
probability of a Type 1 Error (a spurious result). To guard against that, you should consider only using the 
.01 or .001 level of significance. 

For overall comparisons you may find the mean of the Overall Ratings (3.1 to 3.6) for each respondent in 
each group. Then compare the group means using a t-tesl. Means may also be computed for sections of 
the QU1S™T“ or for all of the items on the QUIS'^”™ compared between groups. Or. of course, you 
may make comparisons at the individual item level. But again, beware of Type I Error. 
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More Sophisticated Analyses: 

Th€ number of analyses that one can play with is nearly endless, if you have enoush data points. One 
should be cautious of over-analyzing the data. You will be bound to find something fascinating but 
unreliable. Nevertheless, some interesting additional analyses can be used to investigate the correlational 
structure of the items. These analyses can reveal the underlying importance or relevance of items to the 
users and to overall satisfaction. These include; factor analysis, item analysis, and hierarchical regression 
analysis. 



THIS PAGE LEFT INTENTIONALLY BLANK 


102 



APPENDIX D: SURVEY RESULTS 


A. QUESTIONNAIRE USED FDR EVALUATION 

This questionnaire measured how web-based courses can be rated. The original 
survey was placed on-line, and due to its size, it was unable to be transferred into this 
document. To compensate for this, the questionnaire was restructured to reflect the Web 
version. The survey begins on the following page. 
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The Instructional Design-Usability Questionnaire 


My name is Erich Stefanyshyn and I am a student at the Naval Postgraduate 
School (NPS) conducting research with Professors Ciavarelli and Darken on: 


• Why usability and instructional design principles are not independent 
of each other when building an on-line course. 

• Why the two must be taken into consideration when designing an on¬ 
line course for the web. 

In order to conduct research, I need your input on how your online class was 
presented via the web. 

Background: System usability and instructional quality are dependent upon each 
other in web-based instruction. The user may become dissatisfied with an online course 
because the content is not organized and coherent, even though the computer system is 
running perfectly. If students spend more time wrestling with the presentation of the 
subject material rather than the content itself, they begin to divert their attention from the 
subject material to adjust to the presentation design. As a result, students spend more than 
the usual amount of required time on a course and become frustrated with long distance 
education. 

Likewise, the content and presentation may be great, but if the system/program 
functions poorly (low bandwidth, dead-end links, poor user interface etc), the learning 
process is equally difficult. This defeats the purpose of long distance education that 
allows students to take the course asynchronously. 

Your Help: The above description may (or may not) have happened to you. I 
would like to know what you considered to be a good course presentation and what you 
consider bad course presentation. 

This is where I need you to rate your class with the Web-based Questionnaire. 
The Web-Based questionnaire was designed to include both web design and instructional 
design consideration. The instructional design relates to the instructor teaching the 
course, whereas the web design pertains to how the course was designed. 
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Your input will be evaluated and analyzed to determine how NPS ean improve the 
quality of its web-based instruetion eourses. 

Outline of the Questionnaire 

In order to standardize the web-based instruction at NPS, the questionnaire is 
broken down into four parts: Getting the learner Started section: provides an introduction 
to the course, course syllabus, objectives, and course expectations. A Presenting the 
Subject Material section: presents the actual course lesson(s). An Assessing the Subject 
Material section: pertains to how well the exams, term papers, projects, etc. were 
prepared. An Assessing the Usability of the Courseware section: rates how well the 
program (software) ran. 

There will be a more detailed explanation in each section so you do not have to 
come back to this page. 

Number of questions: 83 (78 multiple choice and five short (optional) essay) 

Time to complete the questionnaire : 20-30 minutes 

Thanks for your valuable time and support! 

DEMOGRAPHICS 

Course 

_ SS3011 - Space Systems Technology and Applications, CDR Higgins 
_ OC2022 - Scientific Fortran Programming, Arlene Guest 
_ MN3384 - Acquisition Production and Quality Management, Mike Boudreau 

Was this course delivered 

a. Completely online (web-based) 

b. Hybrid - some class meetings plus web 

c. VTE (video teleconferencing environment) and on-line 
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PART 1: Getting the Learner Started 

This section provides a ''big picture view"' of the course. It assists students by providing a 
course description, prerequisites for the course, course objectives and goals, navigation of 
the web site, and instructor biography, a course syllabus (of chapters to read, homework 
assignments, tests, etc), and course expectations (of the instructor and student). 

Before continuing with the course, students should be able to understand course 
objectives, syllabus, and expectations, and how to access the course lessons. 

INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 

Not Applicable = NA. 

Instructional Quality 

Course Objectives 


1. 

Course goals defined 

confusing clear 




123456789 

NA 

2. 

Brief course description 

confusing clear 




123456789 

NA 

3. 

Course prerequisites 

confusing clear 




123456789 

NA 

4. 

Students given instructions/orientation 
on course navigation 

unhelpful helpful 




123456789 

NA 

5. 

Biography of instructor 

unhelpful helpful 




123456789 

NA 

Course Svllabus/Class Schedule 



6. 

Class schedule (when class meets) 

confusing clear 




123456789 

NA 

7. 

Class agenda (amount of reading per week, 
required assignments due, exam dates) 

confusing clear 




123456789 

NA 


Course Expectations 

8. Role of the instructor was defined confusing clear 

123456789 NA 
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9. Students’ responsibilities confusing clear 

123456789 NA 

10. Grading criteria and procedure confusing clear 

123456789 NA 

Web Design 

Course Objectives 

11. Content from course objectives 
(i.e. font, bold, italics, underlines, 

misspellings, etc) hard to read easy to read 

123456789 NA 

12. Content provided in understandable 

paragraphs inadequate adequate 

123456789 NA 

13. Content is meaningfully labeled illogical logical 

123456789 NA 

Course Svllabus/Class Schedule 

14 Content (i.e. font, bold, italics, 

underlines, misspellings, etc) hard to read easy to read 

123456789 NA 

15. Content provided in understandable 

paragraphs inadequate adequate 

123456789 NA 

16. Content is meaningfully labeled illogical logical 

123456789 NA 

Course Expectations 

17. Content (i.e. font, bold, italics, 

underlines, misspellings, etc) hard to read easy to read 

123456789 NA 

18. Content provided in understandable 

paragraphs inadequate adequate 

123456789 NA 

19. Content is meaningfully labeled illogical logical 

123456789 NA 

20. Please write your comments about the Getting the Learner Started section here: 
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PART 2: Presenting the Subject Material 

The instructor may have organized the material into weekly folders, topic folders, and/or 
a hyperlink for the week for the student to view or download. 

In this area, the number of options for delivering a lesson ranges from purchasing a 
textbook for the course, additional reading(s) via the web, problems, exercises, 
discussions via chat rooms and/or discussion boards, written assignments (summaries), 
and/or extra web-links to enhance the lesson(s). This section evaluates the actual 
lesson/module. This is where the actual learning begins. 


INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 

Not Applicable = NA. 


Instructional Quality 


Purpose of the Course 


21. 

The purpose of each module/section was 
adequately defined 

never always 

123456789 

NA 

22. 

Learning objectives for each component of 
the course were adequately defined 

never always 

123456789 

NA 

23. 

Weekly/module folders are in logical order 

never always 

123456789 

NA 

Instructor’s Particioation 



24. 

Students were encouraged to collaborate 

never always 

123456789 

NA 

25. 

Instructor gave learning guidance 

never always 

123456789 

NA 

26. 

Learning guidance was adequate 

never always 

123456789 

NA 

27. 

The instructor responded to students’ 
questions/concerns regarding material 

never always 



123456789 NA 
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28. Instructor’s participation met expectations never always 

123456789 NA 


Presentation of Subject Material 


29. 

Grading criteria and procedure 

difficult easy 




123456789 

NA 

30. 

Instructional material was presented at 
proper level 

difficult easy 




123456789 

NA 

31. 

Instructional explanations of materials 

confusing clear 




123456789 

NA 

32. 

Instructional explanations included 
useful illustrative ejamples (pictures, 
graphs, diagrams, etc) 

never always 




123456789 

NA 

33. 

Assigned reading related to course objectives 

never always 




123456789 

NA 

34. 

Instructional content relevant to learning 
objectives 

never always 




123456789 

NA 

35. 

Instructional content in correct sequence 

never always 




123456789 

NA 

36. 

Supplementary web sites provided to 
enhance learning 

never always 




123456789 

NA 

37. 

Subject material was highly interactive 

never always 




123456789 

NA 

38. 

Interaction of subject material useful 

never always 




123456789 

NA 

39. 

Interaction of subject material useful 

never always 




123456789 

NA 

40. 

Content engaged the student in active 
learning tasks 

never always 




123456789 

NA 

Assienments for Subject Material 



41. 

Assignment helped achieve 
learning objectives 

never always 




123456789 

NA 

42. 

Review of material was provided 

never always 




123456789 

NA 
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43. 

Instructional practice examples were provided 

never always 




123456789 

NA 

44. 

Discussion board was used during the course 

never always 




123456789 

NA 

45. 

Discussion board useful 

never always 




123456789 

NA 

46. 

Chat rooms were used during the course 

never always 




123456789 

NA 

47. 

Chat rooms useful 

never always 




123456789 

NA 

48. 

Written assignments (term papers, one-page 
Summaries, etc) were included in each 
assignment 

never always 




123456789 

NA 

Feedback Section for Students 



49. 

Feedback section on lesson available 
for students 

never always 




123456789 

NA 

50. 

Feedback section on lesson useful 

never always 




123456789 

NA 

Web Design 



51. 

Subject material content 

(i.e. font, bold, italics, underlines, 

misspellings, etc) 

hard to read easy to read 




123456789 

NA 

52. 

Subject material content provided in 
understandable paragraphs 

inadequate adequate 




123456789 

NA 

53. 

Content is meaningfully labeled 

illogical logical 




123456789 

NA 

54. 

Please write your comments about the Presentation of Subject Material here: 



no 



PART 3: Assessing the Subject Material 

This section rates the exams, projects, term papers, etc. of the subject material. 
INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 


Not Applicable = NA. 

Instructional Quality 


55. 

Exam questions relevant to learning objectives 

never always 

123456789 

NA 

56. 

Exam questions was presented at proper level 

never always 

123456789 

NA 

57. 

Exam questions are understandable 

never always 

123456789 

NA 

58. 

Exam has a reasonable time limit 

never always 

123456789 

NA 

59. 

Exam provided feedback and review 
when questions answered incorrectly 

never always 

123456789 

NA 

60. 

Instructional content provided adequate 
exam preparation 

never always 

123456789 

NA 

61. 

Student critique section of exam provided 

never always 

123456789 

NA 

62. 

Assessments other than exams (papers, projects, 
assignments, etc) were used in this course) 

never always 

123456789 

NA 

63. 

Assessments relevant to learning objectives 

never always 

123456789 

NA 

Web Design 



64. 

Exam content 




(i.e. font, bold, italics, underlines, misspellings, etc) hard to read easy to read 

123456789 NA 
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65. 


Exam content provided in 

understandable paragraphs inadequate adequate 

123456789 NA 


66. Content is meaningfully labeled illogical logical 

123456789 NA 

67. Please write your comments about the Assessing of Subject Material section here: 


PART 4: Assessing the Usability of the Courseware 

Courseware is the software program that is used to deliver the course (Blackboard, Cnet, 
Courseware, etc) online. If the courseware does not function properly (interface, buttons, 
link time, navigation, runs slow), it may be difficult for the student to concentrate on the 
material. 

Input from students on how the courseware functions, is vital for the success of web- 
based instruction because if students cannot use it, then the courseware hindered the 
learning process. 

INSTRUCTIONS 

Please circle the numbers, which most appropriately reflect your impressions about this 
course. 

Not Applicable = NA. 


68. 

Student critique provided for courseware 

never 

always 



1 2 345 

67 8 9 

69. 

Courseware was easy to learn 

never 

always 



1 2 345 

67 8 9 

70. 

Courseware was easy to use (after learning it, did you 
struggle with it for the remainder of the course) 

never 

always 



1 2 345 

67 8 9 

71. 

Course site was accessible (connectivity, firewall) 

never 

always 



1 2 345 

67 8 9 

72. 

Course site required to much navigation to 
view actual content (clicking until site reached) 

never 

always 



1 2 345 

67 8 9 
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73. 

Were there difficulties in navigating from page to page 

never always 




123456789 

NA 

74. 

Were there difficulties in navigating between lessons 

never always 




123456789 

NA 

75. 

Were there difficulties in navigating from course page 
to web links and back to course page 

never always 




123456789 

NA 

76. 

Were there difficulties in paging from top of page 
to bottom of page 

never always 




123456789 

NA 

77. 

Were there difficulties in paging from bottom of page 
to top of page 

never always 




123456789 

NA 

78. 

Were there difficulties in finding the 
“home” page 

never always 




123456789 

NA 

79. 

Were there difficulties in finding the 
desired web page (location) 

never always 




123456789 

NA 

80. 

Speed of Courseware was adequate (waiting a long 
time to go to next site and for downloads) 

never always 




123456789 

NA 


81. Please write your comments about Assessing the Usability of the Courseware here: 


END OF QUESTIONNAIRE !! 
Thank you for your cooperation!! 
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B. GRAPHING OF RESULTS 

The results from the survey were interpreted measured on Excel spreadsheet and 
then transferred to the program Stat View for graphing. The purpose of the graphs is to 
compare the average of the respondents based on the class conditions. The two conditions 
are: 

• Hybrid (class/web-enhanced) and 

• VTE (video teleconferencing/web-enhanced). 

They offer an opportunity to compare not only two different groups, but also the 
chance of evaluating how well the questionnaire can measure web-based instruction. The 
results show that the questionnaire does deliver constant results and can measure web- 
based instruction. 

The averages were obtained by the rating system provided by the QUIS 7.0 
(APPENDIX C: QUESTIONNAIRE EOR USER INTERACTION SATISEACTION). 
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Graph 1: Summary of Areas 
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Split By: Condition 
Error Bars: ± 1 Standard Deviation(s) 
Row exclusion: Erich2.xis (imported) 
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Graph 2: Getting the Learner Started 
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Row exclusion: Erich2.xls (imported) 
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3. Graph 3: Presenting the Subject Material 


Cell Line Chart 

Split By: Condition 

Row exclusion: Erich2.xls (imported) 



Hybrid 


VTE/On-lin 


117 






4. Graph 4: Assessing the Subject Material 


Cell Line Chart 
Split By: Condition 
Row exclusion: Erich2.xls (imported) 
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5. Graph 5: Assessing the Usability of the Courseware 


Cell Line Chart 
Split By: Condition 
Row exclusion: Erich2.xls (imported) 
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